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COMPTROLLER  GENERAL ' S  BETTER  CONTROLS  NEEDED  OVER 

REPORT  TO  THE  CONGRESS  BIOMEDICAL  RESEARCH  SUPPORTED  BY 

THE  NATIONAL  INSTITUTES  OF  HEALTH 
Department  of  Health,  Education, 
and  Welfare 


DIGEST 

The  National  Institutes  of  Health,  an  agency 
of  the  Department  of  Health,  Education,  and 
Welfare,  is  composed  of  11  institutes  and 
supports  the  majority  of  biomedical  research 
done  in  the  United  States.  In  fiscal  year 
1974  it  obligated  about  $2  billion  for  this 
purpose.  (See  p.  1.) 

GAO  believes  this  report  will  especially 
interest  the  Senate  Committee  on  Appropri- 
tions.  The  Committee  stated  in  its  report 
on  the  Department's  fiscal  year  1976  appro¬ 
priation  request  that  most  of  each  year's 
funds  are  committed  to  ongoing  projects, 
leaving  little  money  to  support  new  projects. 

The  Committee  urged  the  Institutes  to  con¬ 
sider  whether  a  different  balance  between 
old  and  new  projects  was  desirable  and,  if 
so,  how  it  could  be  accomplished  without  un¬ 
duly  disrupting  productive  ongoing  research. 

Grantees  do  most  of  the  research  funded  by 
the  Institutes.  Grantees  compete  for  fund¬ 
ing  on  the  basis  of  their  grant  applica¬ 
tion's  scientific  merit,  which  is  deter¬ 
mined  by  authorities  in  selected  scientific 
fields.  Approved  grantees  generally  re¬ 
ceive  funding  for  3  years.  After  the  ap¬ 
proval  period  expires,  applications  for 
continued  financial  support  are  treated  as 
new  grant  applications  and  grantees  must 
again  compete  for  funding. 

At  the  three  institutes  GAO  reviewed, 
nearly  all  noncompeting  grants  received 
funding  for  each  year  of  the  approved 
period,  but  funding  was  available  for  only 
34  to  50  percent  of  competing  grants.  The 
Institutes  should  reassess  noncompeting 
grants  annually  to  insure  that  continued 
funding  is  desirable,  because: 
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— Many  unfunded  competing  grant  applications 
nad  greater  scientific  merit,  as  evidenced 
by  the  priority  scores  assigned  by  scien¬ 
tific  authorities,  than  some  noncompeting 
grants  which  continued  to  be  funded. 

— About  44  percent  of  the  noncompeting  grants 
were  not  funded  again’  after  their  approval 
period  expired  because,  when  competing  with 
other  grant  applicants,  their  priority 
scores  indicated  they  were  of  lower  scien¬ 
tific  merit. 

The  Institutes  does  not  terminate  a  research 
grant  when  progress  is  poor  and  cannot,  under 
existing  regulations,  terminate  when- 

--significantly  better  applications,  as 
evidenced  by  priority  scores,  could  be 
funded  or 

— recent  scientific  developments  or  other 
events  result  in  research  being  no  longer 
of  public  benefit.  (See  p.  9.) 

Even  if  program  administrators  identified  on¬ 
going  research  that  they  believed  to  be  un¬ 
necessary  or  duplicative,  these  grants  could 
not  be  terminated.  Although  the  Institutes 
can  withhold  funds  from  a  grantee,  grant  ad¬ 
ministrators  differ  on  when  this  can  be  aone . 

( See  p.  10  .  ) 

The  Institutes  requires,  as  a  condition  of  a 
grant  award,  that  it  receive  certain  reports 
from  grantees  at  the  end  of  a  research  grant 
period.  These  reports  were  not  being  sub¬ 
mitted  for  many  research  grants.  (See 
p.  21.)  In  addition,  some  grantees  which 
had  not  met  the  reporting  requirements  under 
a  previous  grant  were  being  funded  by  the 
Institutes  under  another  grant.  (See  p.  23.) 

In  contrast  to  extensive  peer  review  of  re¬ 
search  proposals  in  the  Institutes'  extra¬ 
mural  research  grant  programs,  the  Institutes 
had  no  formal  procedures  for  initiating  new 
intramural  research  projects  (those  conducted 
by  Institutes  scientists  in  Institutes  labora¬ 
tories).  ( See  p .  27.) 


Individual  projects  normally  received  only  the 
approval  of  the  branch  or  laboratory  chief 
where  the  research  was  to  be  done.  Reviews 
of  researchers'  work  by  advisory  groups  within 
the  Institutes  concluded  that  some  were  doing 
work  of  questionable  scientific  merit  or  rele¬ 
vance  to  an  institute's  mission.  They  recom¬ 
mended  terminating  the  work  or  taking  other 
corrective  action.  (See  p.  29.) 

GAO  is  making  a  number  of  recommendations  to 
the  Secretary  of  the  Department  of  Health, 
Education,  and  Welfare  to  help  insure  the 
Institutes  support  research  proposals  with 
the  greatest  scientific  merit.  The  recommen¬ 
dations  include 

— developing  a  system  to  identify  noncompeting 
grants  with  significantly  less  scientific 
merit  than  unfunded  competing  grant  applica¬ 
tions  ( see  p  .  19 ) , 

— changing  the  regulations  to  allow  termina¬ 
tion  of  research  grants  under  certain  con¬ 
ditions,  such  as  when  applications  have  sig¬ 
nificantly  greater  scientific  merit,  as  evi¬ 
denced  by  priority  scores  (see  p.  19),  and 

establishing  a  formal  system  for  approving 
all  intramural  research  projects.  (See 
p.  32 . ) 

The  Department  generally  disagrees  with  these 
recommendations,  stating  they  would  be  ex¬ 
tremely  costly  with  little  prospect  of  ap¬ 
preciable  benefits. 

After  evaluating  the  Department's  comments, 

GAO  continues  to  believe  that  implementing 
the  recommendations  would  improve  the  re¬ 
search  of  the  National  Institutes  of  Health. 
The  Department  could  implement  the  recom¬ 
mendations  in  a  simpler  and  more  effective 
manner  than  indicated  in  its  comments,  with 
a  minimum  of  additional  cost. 

GAO  is  also  making  other  recommendations  to 
the  Department  for  improving  management  prac¬ 
tices  with  which  it  generally  agreed. 
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CHAPTER  1 


INTRODUCTION 

Today,  the  National  Institutes  of  Health  (NIH) ,  Depart¬ 
ment  of  Health,  Education,  and  Welfare  (HEW),  is  one  of  the 
world's  foremost  and  prestigious  biomedical  research  centers 
and  is  the  focal  point  for  Federal  biomedical  research  and 
research  support.  The  mission  of  NIH  is  to  improve  the  health 
of  the  people  of  the  United  States.  To  achieve  this  it: 

--Conducts  biomedical  research  in  its  own  labora¬ 
tories  . 

--Provides  grants  to  nonprofit  organizations  and 
institutions  for  research  and  for  medical 
education,  including  improvements  and  construction 
of  library  facilities,  buildings,  equipment, 
and  other  resources. 

--Provides  grants  for  training  research  investi¬ 
gators  . 

--Awards  research  contracts  to  profit  and  nonprofit 
institutions . 

--Supports  biomedical  communications  through  programs 
and  activities  of  the  National  Library  of  Medicine. 

NIH  has  11  research  institutes;  each  supports  biomedical 
research  programs.  All  but  1  institute  are  located  on  the 
NIH  reservation  in  Bethesda,  Maryland,  where  NIH  also  main¬ 
tains  hundreds  of  laboratories,  a  516-bed  clinical  center 
(research  hospital),  and  the  National  Library  of  Medicine. 

The  National  Institute  of  Environmental  Health  Sciences 
(NIEHS)  is  located  at  Research  Triangle  Park,  North  Carolina. 

PROGRAM  ACTIVITY 


Research  conducted  by  NIH  scientists  in  NIH  laboratories 
is  referred  to  as  the  intramural  program;  research  performed 
under  NIH  grants  and  contracts  is  referred  to  as  the  extra¬ 
mural  program. 

In  fiscal  year  1974  NIH  obligated  nearly  $2  billion  to 
support  its  research  activities.  Approximately  $1.6  billion 
of  this  amount  was  used  to  support  research  programs,  and 
the  remainder  was  used  for  research  training,  construction, 
and  overall  management  of  NIH  programs.  In  fiscal  year  1975 
NIH  was  appropriated  $2.1  billion  and  paid  about  38  percent 
of  the  cost  of  all  health  related  research  and  development 
done  in  the  United  States. 
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The  chart  below  shows  NIH's  total  obligations  in  fiscal 
year  1974  and  the  obligations  of  the  three  institutes  we 
reviewed:  the  National  Institute  of  Child  Health  and  Human 

Development  (NICHD) ,  the  National  Institute  of  Allergy  and 
Infectious  Diseases  (NIAID) ,  and  NIEHS. 

NIH  NIAID  NICHD  NIEHS 

- ( millions ) - 


Research  grants 

$1; 

,066 

$71 

$84 

$16 

Intramural  research 

134 

21 

14 

8 

Research  and  development 

335 

12 

21 

2 

contracts 

Research  training 

210 

11 

17 

5 

Program  management 

97 

4 

7 

1 

Other  programs  and  support 

152 

2 

1 

— 

Total 

$1, 

,994 

$121 

$144 

$32 

During  fiscal  year  197 

4, 

about 

70  percent 

of  the 

research 

grants  awarded  by  these  three  institutes  were  for  traditional 
research  grants,  which  are  awards  to  institutions,  in  the 
name  of  a  principal  investigator,  for  discrete  research  proj¬ 
ects.  The  balance  of  the  grant  funds  was  for  other  types  of 
grants,  such  as  program  project  grants,  which  support  broadly 
based,  long-term  activities  involving  teams  of  scientists. 

In  fiscal  year  1974  NIH  received  about  10,000  competing 
research  grant  applications.  Of  these  applications,  74  per¬ 
cent  were  approved  for  funding,  but  funds  were  available  for 
only  53  percent  of  the  approved  applications. 

In  recent  years  NIH  has  experienced  a  widening  gap  be¬ 
tween  the  number  of  research  grant  applications  approved  and 
those  it  can  fund.  As  a  consequence  of  increased  competition 
for  research  funds  coupled  with  budget  constraints,  NIH  must 
be  able  to  identify  and  fund  those  projects,  both  extramural 
and  intramural,  having  the  greatest  scientific  merit  and 
relevance  to  its  mission 

GRANTS  FOR  EXTRAMURAL  RESEARCH 


First  stage 


Scientists  in  the  NIH  Division  of  Research  Grants  read 
an  application  for  a  research  grant  and  assign  it  to  the 
institute  responsible  for  supporting  research  in  that 
scientific  area.  At  the  same  time,  they  assign  each  appli¬ 
cation  to  1  of  approximately  50  study  sections,  or  initial 
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review  groups,  which  reviews  the  application  for  scientific 
merit.  This  review  by  a  study  section  is  the  first  stage  of 
NIH's  dual  review  process. 

Most  study  sections  have  12  to  16  members--authorities 
in  their  selected  scientific  fields.  Each  application  is 
reviewed  and  discussed  before  receiving  the  study  section's 
recommendation  to  approve,  disapprove,  or  defer  it. 

The  study  section,  as  a  whole,  assigns  a  numerical 
priority  score  to  each  approved  application  on  the  basis  of 
its  scientific  merit  relative  to  the  "state  of  the  art"  of 
a  particular  research  area.  A  priority  score  of  between  100 
and  500,  which  represents  an  average  of  the  individual  scores 
given  by  study  section  members,  is  assigned  to  each  appli¬ 
cation.  The  priority  score  indicates  the  relative  scientific 
merit  of  the  grant  application,  and  the  research  institutes 
use  it  to  rank  a  grant  application  for  funding.  The  lower 
the  priority  score,  the  greater  the  scientific  merit  of  the 
application.  A  study  section  can  disapprove  a  grant  appli¬ 
cation  for  a  number  of  reasons,  one  of  the  most  important 
being  lack  of  scientific  merit. 

The  Division  of  Research  Grants  adjusts  the  actual  or 
"raw"  scores  assigned  to  approved  grants  by  study  sections. 
This  adjustment  (normalization)  is  made  to  account  for  scor¬ 
ing  biases  in  each  study  section  and  is  intended  to  place 
the  scores  of  all  study  sections  on  the  same  scale  so  a  given 
score  will  be  comparable  from  one  study  section  to  another. 

Second  stage 


Following  the  study  section  recommendations,  all  approved 
and  disapproved  grant  applications  are  forwarded  to  the 
national  advisory  council  of  the  appropriate  institute.  Each 
institute  has  a  national  advisory  council,  which  must  approve 
grant  applications  before  they  can  be  funded.  These  councils 
are  the  second  stage  of  the  dual  review  process  and  consist 
of  approximately  12  to  18  members— leader s  in  fundamental  and 
medical  science,  education,  and  public  affairs. 

While  these  advisory  councils  may  review  the  scientific 
merit  of  a  grant  application,  their  primary  responsibility 
is  to  evaluate  whether  applications  relate  to  the  mission  and 
the  needs  of  the  respective  institutes.  They  may  either  agree 
or  disagree  with  the  study  section  recommendations. 

After  the  advisory  council  review,  the  institutes  con¬ 
sider  all  approved  applications  for  funding.  Although  the 
institutes  may  fund  grant  applications  out  of  numerical  order, 
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this  is  the  exception  rather  than  the  rule.  In  addition,  each 
institute  receives  the  raw  and  the  adjusted  priority  scores 
for  each  grant  application. 

Grant  applications  the  institutes  approve  for  funding 
are  funded,  on  the  average,  for  3  years.  During  this  3-year 
period,  a  grantee  does  not  have  to  compete  for  funds,  and 
the  grant  is  referred  to  as  a  noncompeting  continuation 
research  grant.  If,  at  the  end  of  an  initially  approved 
period,  a  grantee  requests  additional  funding,  it  must  again 
compete  for  available  funds. 

INTRAMURAL  RESEARCH 

NIH  scientists  conduct  intramural  research  projects  in 
NIH  laboratories  or  the  NIH  clinical  center.  In  contrast  to 
extramural  research  grants,  intramural  research  projects 
receive  no  peer  review  before  they  are  begun. 

Intramural  projects  at  the  institutes  reviewed  are 
directed  by  either  the  scientific  director  within  each 
institute  or  the  director  of  the  institute  who  allots  space 
and  resources  to  individual  research  branches.  Branch  or 
section  chiefs  generally  approve  individual  projects,  subject 
to  review  by  the  scientific  director. 

In  fiscal  year  1974,  NIH  obligated  $134  million  for  intra 
mural  research.  Also  in  fiscal  year  1974,  NIAID ,  NICHD , 
and  NIEHS  obligated  a  combined  total  of  $43  million  for  intra¬ 
mural  research,  15  percent  of  their  combined  budgets  of  $297 
million.  This  amount  accounted  for  32  percent  of  the  total 
NIH  obligations  for  intramural  research. 

NIH  officials  said  intramural  research: 

--Allows  administrative  barriers  between  medical 
disciplines  to  be  crossed,  which  cannot  be  done 
at  medical  schools. 

--Permits  a  large  number  of  medical  skills  to  be 
brought  together  in  one  place,  so  quality  bio¬ 
medical  research  can  be  done. 

--Is  conducive  to  spontaneous,  collaborative 
research  efforts  among  scientists. 

Intramural  research  also  enables  scientists  to  devote 
most  of  their  time  to  their  research,  while  under  an  extra¬ 
mural  research  grant  a  researcher's  time  is  divided  among 
research,  teaching,  and  various  administrative  duties. 
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CHAPTER  2 


NEED  FOR  BETTER  APPROVAL  PROCEDURES 
AND  IMPROVED  MONITORING  OF  GRANTS 

The  National  Institutes  of  Health  research  institutes  we 
reviewed  funded  noncompeting  multiyear  research  grants  during 
the  period  for  which  they  were  initially  approved,  without 
comparing  their  scientific  merit  with  that  of  competing  grant 
applications.  However,  many  unfunded  competing  grant  applica¬ 
tions  had  greater  scientific  merit,  evidenced  by  their  prior¬ 
ity  scores,  than  some  noncompeting  grants  that  continued  to 
be  funded.  About  44  percent  of  the  noncompeting  grants  were 
not  funded  after  their  approval  period  expired,  and  they 
again  had  to  compete  for  funds. 

Continuing  the  funding  of  some  noncompeting  continuation 
grants  may  be  desirable,  even  if  their  scientific  merit  may 
be  less  than  unfunded  competing  applications.  They  should 
be  reassessed  before  denying  funds  to  new  applications  with 
significantly  greater  scientific  merit.  This  cannot  be  done, 
partly  because  NIH  does  not  have  written  guidelines  deline¬ 
ating  the  type  of  review  or  specific  tasks  grant  administra¬ 
tors  must  perform  during  their  review  of  ongoing  research. 

If  a  grantee  has  complied  with  the  terms  and  conditions 
of  a  grant  award,  NIH  cannot  terminate  the  grant  for  the 
convenience  of  the  Government  without  the  grantee's  concur¬ 
rence.  Although  NIH  can  withhold  funds  from  a  grantee  for 
the  remainder  of  an  approved  project  period,  it  cannot 
rescind  funds  that  were  approved  but  unspent  for  the  earlier 
part  of  the  grant  period.  In  addition,  grant  administrators 
differ  on  when  funds  should  be  withheld  from  a  grantee. 

NIH  does  not  require  the  11  institutes  to  use  normalized 
priority  scores  in  funding  grants.  Therefore,  NIH  has  no 
assurance  that  applications  with  the  greatest  scientific  merit 
are  being  funded.  Some  grants  probably  would  not  have  been 
funded  if  normalized  scores  were  used,  and  some  would  have 
been  funded  but  were  not. 

UNFUNDED  COMPETING  APPLICATIONS 
WITH  SCIENTIFIC  MERIT  GREATER 
THAN  NONCOMPETING  GRANTS 

At  the  three  NIH  institutes  reviewed,  nearly  all  ($63.9 
million  worth)  of  the  noncompeting  multiyear  traditional 
research  grant  applications  were  funded  in  fiscal  year  1974. 
During  the  same  period,  these  institutes  were  only  able  to 
fund  between  34  and  52  percent  of  approved  competing 
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traditional  research  grant  applications  for  a  total  of  $45.1 
million.  The  chart  below  shows  the  funding  of  competing  and 
noncompeting  grants  for  the  three  institutes  for  fiscal  year 
1974  . 


Institutes 

Noncompeting 

grants 

Amount  Number 

Competing 

grants 

Amount  Number 

Percent  of 
approved 
competing 
grants 
funded 

National  Institute  of 
Environmental  Health 
Sciences 

(millions ) 

$  3.5 

80 

(millions ) 

$  2.4 

37 

34 

National  Institute  of 
Child  Health  and 

Human  Development 

25.1 

538 

26.7 

469 

52 

National  Institute  of 
Allergy  and  Infec¬ 
tious  Diseases 

35.3 

704 

16.0 

327 

46 

Total 

$63.9 

1,322 

$45.1 

833 

A  comparison  of  the  normalized  priority  scores  of 
approved,  unfunded  grant  applications  with  the  normalized 
priority  scores  of  funded  noncompeting  multiyear  grants  for 
NICHD  and  NIAID  showed  that  during  fiscal  year  1974,  287 
unfunded  research  grant  applications  had  priority  scores 
equal  to  or  better  than  the  noncompeting  grants  that  were 
funded. 1  An  NIH  official  indicated  that  the  normalization 
process  generally  permits  priority  scores  to  be  compared 
validly  from  one  year  to  the  next. 

In  NIAID  144  competing  grants  were  not  funded,  although 
they  had  equal  or  better  priority  scores  than  20  noncompeting 
grants  that  were  funded.  In  NICHD,  143  competing  grants  that 
were  not  funded  had  equal  or  better  priority  scores  than  106 
noncompeting  grants  that  were  funded. 

Grant  administrators  are  responsible  for  annually  review¬ 
ing  ongoing  research  grants  for  scientific  progress,  budgetary 
matters,  and  overall  direction  of  research.  The  grant 
administrators  we  spoke  with  do  not  make  critical  reviews  of 
noncompeting  research  grants  to  identify  ongoing  grants  which 
might  have  less  scientific  merit  or  public  benefit  than  com¬ 
peting  grants  that  are  approved  but  unfunded. 

The  need  for  NIH  to  continually  assess  the  expected 
benefits  of  funding  a  noncompeting  multiyear  research  grant 
is  also  evident  from  the  number  of  grants  that  are  not 


1 


NIEHS  does  not  use  normalized 


scores . 
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funded  after  the  grant  period  expires.  During  fiscal  year 
1974,  the  three  institutes  did  not  fund  44  percent  (223) 
of  the  multiyear  traditional  research  grants  whose  terms  had 
expired  and  for  which  renewal  applications  had  been  submitted. 
Of  these  grants,  180  were  approved,  but  they  did  not  have 
sufficient  scientific  merit  to  warrant  support  considering 
funding  limitations  of  the  respective  institutes.  NIH  study 
sections  disapproved  the  remaining  43  research  grants. 

The  following  table  shows  the  number  of  competing  re¬ 
newal  research  grants  that  were  approved  and  funded,  approved 
and  unfunded,  and  disapproved  by  each  of  the  three  institutes 
in  fiscal  year  1974. 

Approved- funded  Approved- unfunded  Disapproved  Total _ 


No 

Percent 

No. 

Percent 

No. 

Percent 

No. 

Percent 

NIEHS 

15 

42 

16 

44 

5 

14 

36 

100 

NIAID 

132 

56 

91 

38 

15 

6 

238 

100 

NICHD 

133 

58 

73 

32 

23_ 

10 

229 

100 

Total 

280 

56 

180 

36 

43 

8 

503 

100 

The  reasons  NIH  study  sections  gave  for  disapproving  the 
43  grant  applications  shown  above  included  (1)  lack  of  detail 
in  the  grant  proposal,  (2)  lack  of  research  technique,  (3) 
poor  research  design,  and  (4)  poor  progress  under  previous 
grant  awards.  For  the  3  institutes,  14  grant  applications 
were  disapproved  primarily  on  the  basis  of  poor  research 
progress  in  prior  years.  NIH  had  supported  these  14  research 
grants  for  an  average  of  3  years.  The  individual  grants  had 
run  as  long  as  6  years,  and,  in  total,  NIH  awarded  grant  funds 
of  at  least  $1.1  million  to  support  these  grants. 

We  reviewed  the  priority  scores  of  the  16  approved  and 
unfunded  competing  renewal  applications  for  NIEHS,  the  91 
for  NIAID ,  and  the  73  for  NICHD.  NIEHS  funded  grants  during 
fiscal  year  1974  with  priority  scores  of  212  or  lower,  and 
10  of  the  16  approved  unfunded  competing  renewals  had  scores 
of  300  or  more.  NIAID  funded  grants  with  scores  through  249, 
and  32  of  the  91  approved  unfunded  competing  renewals  had 
scores  of  300  or  more.  NICHD  had  22  of  its  73  applications 
with  normalized  priority  scores  of  300  or  more. 

LACK  OF  GUIDANCE  FOR 
REVIEW  OF  ONGOING  RESEARCH 


After  a  research  grant  has  been  funded,  the  NIH  grant 
administrator,  who  has  a  scientific  background,  is  to  pro¬ 
vide  overall  surveillance  and  management  until  the  termination 
of  the  grant.  Ongoing  research  is  monitored  primarily  through 
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(1)  annual  progress  reports,  which  accompany  the  continuation 
application  for  a  noncompeting  award,  (2)  publications  result¬ 
ing  from  the  supported  research,  and  (3)  personal  contact  with 
the  grantee.  The  progress  report  is  the  most  important 
instrument  in  monitoring  ongoing  research.  Progress  reports 
for  ongoing  research  are  reviewed  to  determine  (1)  significant 
research  developments  and  (2)  departures  from  the  original 
technical  objectives  or  unusual  circumstances  that  could 
impede  the  progress  and  successful  completion  of  the  re¬ 
search  . 

No  written  guidelines  exist,  however,  to  delineate 
specific  tasks  or  functions  to  be  done  by  grant  administrators 
during  their  review  of  annual  progress  reports  for  ongoing 
research.  Grant  administrators  said  their  review  of  these 
reports  was,  for  the  most  part,  discretionary,  resting  on 
their  professional  judgment. 

Although  the  grant  administrators  at  the  three  institutes 
appeared  to  be  making  similar  reviews,  we  could  not  evaluate 
the  extent  or  intensity  of  the  reviews,  because  grant  adminis¬ 
trators  are  not  required  to  document  them  or  make  formal 
written  comments  about  the  progress  of  the  research. 

One  institute  director  said  progress  reports  often  do 
not  contain  sufficient  information  to  enable  a  meaningful 
evaluation  of  ongoing  research.  Some  progress  reports  are 
adequately  prepared,  while  others  may  lack  sufficient 
technical  information  to  assess  progress  by  grantees.  One 
grant  administrator  explained  that  (1)  instructions  for 
preparing  progress  reports  are  not  specific  and  (2)  grantees 
spend  a  minimum  of  time  in  preparing  these  reports.  Most 
grant  administrators  we  talked  with  said  the  length,  detail, 
and  completeness  of  progress  reports  vary  among  grantees. 

If  NIH  grant  administrators  critically  reviewed  non¬ 
competitive  multiyear  research  grants  with  respect  to  their 
current  scientific  merit  and  research  progress,  the  funds 
from  some  noncompetitive  grants  with  poor  progress  might  have 
been  withheld  and  applied  to  competing  unfunded  grant  appli¬ 
cations.  For  instance,  in  NIAID  alone,  $134,000  could  have 
been  directed  to  other  projects  in  fiscal  year  1974  if  the 
funding  of  five  research  grants,  which  ultimately  were  dis¬ 
approved  by  the  study  sections  due  to  poor  progress,  had 
been  withheld  during  the  last  year  of  support.  Of  course, 
additional  funds  would  accrue  the  earlier  action  is  taken  to 
withhold  grant  funds. 
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TERMINATION  OF  RESEARCH  GRANTS 


NIH  can  terminate  research  grants  under  the  Code  of 
Federal  Regulations  (45  CFR  74.114),  which  states: 


"the  granting  agency  may  terminate  any  grant  in 
whole,  or  in  part,  at  any  time  before  the  date 
of  completion,  whenever  it  is  determined  that 
the  grantee  has  materially  failed  to  comply  with 
the  terms  and  conditions  of  the  grant." 


Otherwise,  NIH 
of  the  grantee, 
search  grant  at 


can  only  terminate  a 
A  grantee,  however 
any  time. 


grant  with  the 
may  terminate 


consent 
a  re- 


Failure  to  comply  with  the  terms  and  conditions  of  a  re¬ 
search  grant  includes  (1)  performing  research  not  originally 
approved,  (2)  misuse  of  Government  funds,  and  (3)  the  prin¬ 
cipal  researcher  leaving  the  institution  to  which  the  grant 
was  originally  awarded.  We  were  also  advised  that  NIH  cannot 
terminate  a  research  grant  due  to  poor  research  progress.  As 
a  result,  neither  the  Department  of  Health,  Education,  and 
Welfare  nor  NIH  terminates  a  research  grant  or  revokes  the 
funding  of  a  research  grant,  in  the  interest  of  the  Govern¬ 
ment,  except  as  stated  above. 


An  HEW  legal  opinion  in  1969  stated  that  "there  is  no 
way  to  require  a  grantee  not  to  expend  funds  already 
granted  *  *  *"  unless  the  grantee  has  done  something  wrong. 
This  document  also  pointed  out  "that  because  of  the  project 
period  concept,  funds  unexpended  *  *  *  [in  one  fiscal  year] 
would  still  be  available  for  expenditure  in  a  subsequent 
fiscal  year  within  the  approved  project  period."  According 
to  an  NIH  official  responsible  for  policies  and  procedures 
for  research  grants,  this  means  NIH  cannot  terminate  a  grant 
for  poor  research  progress  and  cannot  rescind  funds  already 
awarded  to  a  grantee,  even  though  poor  research  progress  is 
evident.  Although  NIH  may  choose  not  to  award  future  funds 
under  a  research  grant,  he  stated  NIH  cannot  reduce  the 
approved  project  period  during  which  previously  awarded 
funds  can  be  obligated  by  the  grantee. 

Under  existing  regulations,  NIH  cannot  terminate  research 
grant  support,  although  continuation  of  a  project  may  not 
reasonably  be  expected  to  benefit  the  public.  For  example, 
if  a  cure  for  a  disease  were  found  or  other  research  break¬ 
throughs  occurred,  NIH  could  not  end  grant  support  of  projects 
having  similar  objectives  before  their  normal  expiration. 

In  1969  NIH  attempted  to  amend  the  Federal  regulations 
governing  research  grants  to  include  additional  authority  to 
terminate  research  grants  in  such  cases.  It  provided  that 
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the  Secretary  of  HEW  could  terminate  a  grant  whenever  he 
determined  that  the  advance  of  medical  or  scientific  knowl¬ 
edge,  the  progress  of  medicine  or  health-related  sciences, 
or  national  health  needs  were  such  that  a  project's  con¬ 
tinuation  could  no  longer  contribute  to  these  purposes. 

The  intent  of  the  proposed  amendment  was  to  recognize 
that  "today's  research  may  be  rendered  obsolete  by  tomorrow's 
discoveries  and  that  the  necessity  of  health-related  research 
varies  in  accordance  with  *  *  *  the  state  of  the  public 
health."  Although  NIH  desired  to  incorporate  this  provision 
into  the  Federal  regulations,  it  was  rejected  at  the  secre¬ 
tarial  level  of  HEW  in  1970  as  being  detrimental  to  the 
security  and  stability  of  grantees.  NIH  still  cannot  termi¬ 
nate  research  grants  solely  on  the  basis  of  public  interest 
or  convenience  of  the  Government. 

Although  the  1969  legal  opinion  indicated  that  NIH  could 
withhold  future  funds  from  a  grantee,  grant  administrators 
at  the  three  institutes  reviewed  differed  as  to  what  action 
they  would  take  regarding  poor  progress  and  whether  or  not 
NIH  had  a  legal  obligation  to  continue  funding  a  grantee  for 
the  entire  project  period. 

For  example,  one  grant  administrator  at  NIEHS  said 
documents  sent  to  a  grantee,  such  as  the  notice  of  grant 
award  and  policy  statements  on  early  terminations,  make  the 
institute  morally  liable  to  fund  research  grants  for  the 
approved  project  period.  Another  official  at  the  same  insti¬ 
tute  believed  NIH  was  legally  obligated  to  continue  funding 
a  grantee  for  the  entire  project  period.  However,  some 
grant  administrators  at  NICHD  indicated  that  they  would 
initiate  remedial  action,  such  as  withholding  funding,  against 
a  grantee  if  poor  research  progress  were  evident.  At  NIAID, 
we  were  told  that  leverage  would  not  be  used  against  a  grantee 
once  funds  had  been  awarded.  Some  grant  administrators 
believed  the  Government's  only  recourse  for  these  grants  was 
to  disapprove  them  when  they  again  competed  for  funds. 

In  contrast  to  NIH's  policy  of  not  terminating  research 
grants,  two  other  research  organizations  we  visited — the 
Department  of  the  Army  Research  office  and  the  North  Carolina 
Science  and  Technology  Research  Center- -will  terminate  on¬ 
going  grants  if  it  is  in  their  best  interest.  For  example, 
one  organization  terminated  an  ongoing  grant  because  it  found 
that  two  grantees  were  doing  the  same  research.  Officials  in 
both  organizations  believed  that  a  policy  of  not  terminating 
research  grants,  to  protect  the  security  and  stability  of 
grantees,  can  only  be  a  disadvantage  to  the  granting  organiza¬ 
tion.  Department  of  Army  regulations  provide  that  the  Govern¬ 
ment  can  terminate  a  grant  and  that  the  grantee  will  repay  the 
Government  the  uncommitted  balance  of  all  grant  funds  awarded. 
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NEED  FOR  CONSISTENT  RANKING  OF 
COMPETING  RESEARCH  GRANT  APPLICATIONS 


After  the  NIH  study  sections  review  a  competing  research 
grant  application  and  assign  a  priority  score  on  the  basis  of 
scientific  merit,  the  research  institutes  use  either  the 
raw  score  or  normalization  techniques  for  selecting  which 
applications  should  be  funded.  Normalization  is  based  on  the 
belief  that  different  study  sections  have  a  tendency  to 
consistently  assign  relatively  good  or  relatively  poor 
priority  scores,  and  the  raw  scores  are,  in  effect,  normalized 
to  offset  this  tendency. 

Because  a  grant  application  is  funded  depending  on  its 
relative  ranking  with  applications  from  all  study  sections, 
an  NIH  advisory  committee  considered  normalization  a  necessity 

In  1973  the  normalization  evaluation  committee,  which 
evaluated  the  normalized  scoring  system,  recommended  that  (1) 
the  normalized  score  be  adopted  as  the  official  NIH  score  and 
(2)  only  this  score  be  used  for  ranking  applications  reviewed 
by  study  sections. 

At  the  time  of  our  review,  both  the  raw  and  normalized 
scores  were  being  used  to  rank  research  grant  applications . 

Six  of  the  11  NIH  institutes  were  using  normalized  scores, 
while  the  other  5  were  using  the  raw  scores.  Even  though  the 
normalization  evaluation  committee  pointed  out  that  using 
normalized  scores  could  not  be  effective  as  long  as  some 
major  awarding  components  were  not  using  it,  NIH  has  not 
required  its  institutes  to  adopt  the  normalized  scoring  system 

The  committee  also  pointed  out  that,  although  only  a 
few  applications  would  actually  experience  a  change  in  funding 
status,  the  fact  that  one  application  is  funded  at  the  ex¬ 
pense  of  another  cannot  be  regarded  as  trivial.  While  a 
grant  application  might  be  funded  if  normalized  priority 
scores  were  used,  it  might  not  be  funded  if  raw  scores  were 
used.  For  example,  at  NIEHS,  raw  priority  scores  are  used 
to  rank  grant  applications  for  funding.  Of  81  grant  appli¬ 
cations  with  priority  scores  50  points  higher  and  50  points 
lower  than  the  cutoff  score  necessary  for  funding  by  NIEHS 
in  fiscal  years  1973  and  1974,  6  applications  totaling 
$291,000  were  not  funded  on  the  basis  of  raw  scores  but  would 
have  been  funded  if  normalized  priority  scores  had  been  used. 
Instead,  five  other  applications  totaling  $243,000  were  funded 
Although  these  five  applications  had  greater  scientific  merit 
on  the  basis  of  raw  priority  scores,  they  had  less  scientific 
merit  when  normalized  scores  were  used  as  the  basis  for 
evaluation . 
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CONCLUSIONS 


NIH  diligently  tries  to  insure  that  the  scientific  merit 
of  each  grant  application  is  assessed  before  funds  are  com¬ 
mitted  to  a  project. 

In  our  opinion,  NIH  could  better  assure  that  the  grants 
being  funded  were  those  with  the  greatest  scientific  merit 
if  a  system  were  developed  to  identify  noncompeting  grants 
having  less  scientific  merit  than  approved  unfunded  competing 
grant  applications.  This  system  could  use  normalized  priority 
scores.  Once  a  system  is  developed  to  identify  noncompeting 
grants  with  less  scientific  merit  than  approved  unfunded 
competing  applications,  noncompeting  grant  funding  should  not 
be  automatically  discontinued,  since  other  factors  may  make 
it  desirable  to  continue  the  funding.  We  believe,  however, 
that  the  advantages  and  disadvantages  of  continuing  to  fund 
such  noncompeting  grants  should  be  assessed  before  funds  are 
denied  to  applications  with  greater  scientific  merit. 

For  such  a  system  to  be  effective,  NIH  will  also  have 
to  require  all  institutes  to  use  normalized  priority  scores 
in  funding  grant  applications  for  scores  to  be  compared  from 
one  year  to  another.  In  addition,  by  permitting  institutes 
to  fund  grants  on  the  basis  of  raw  priority  scores,  NIH  cannot 
assure  that  the  biases  of  the  individual  study  sections  are 
being  eliminated  and  that  competing  grant  applications  with 
the  greatest  scientific  merit  are  being  funded. 

Also,  the  regulations  should  be  changed  to  permit  grants 
to  be  terminated  in  cases  when  better  applications  could  be 
funded  or  the  research  is  no  longer  of  public  benefit.  In 
addition  the  regulations  should  require  the  grantee  to  make 
reasonable  progress  as  a  material  condition  of  grant  per¬ 
formance  . 

Federal  regulations  preclude  the  termination  of  multi¬ 
year  research  grants,  except  when  the  grantee  has  not  complied 
with  the  terms  of  the  grant.  Because  of  this  policy,  even 
if  grant  administrators  were  to  identify  ongoing  research 
they  believed  to  be  unnecessary  or  duplicative,  the  grants 
could  not  be  terminated. 

We  recognize  that,  except  for  not  being  able  to  rescind 
funds  that  were  approved  in  the  early  part  of  the  grant 
period,  withholding  funds  and  actually  terminating  funds 
differ  very  little.  Grant  administrators  differed,  however, 
on  NIH's  obligation  to  a  grantee  during  the  grant  period. 

A  change  in  the  regulations  would  clarify  and  resolve  NIH's 
obligation  to  grantees. 

NIH  does  not  have  specific  instructions  for  grant 
administrators  who  annually  review  noncompeting  grants.  The 
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type  ana  extent  of  the  review  given  depends  on  the  individual 
grant  administrators'  professional  judgments.  In  addition, 
did  does  not  provide  specific  guidelines  to  grantees  on  what 
should  Pe  incluaed  in  the  annual  scientific  progress  reports 
submitted  to  NIH.  we  believe  that  formulating  such  instruc¬ 
tions  will  enhance  both  the  quality  and  depth  of  the  annual 
reviews  by  NIB  grant  administrators. 

We  proposed  that  the  Secretary  of  HEW  change  the  regu¬ 
lations  for  terminating  research  grants  to  permit  selective 
termination  of  grants  when  (1)  progress  is  poor,  (2)  better 
applications  could  be  funded,  or  (3)  the  research  is  no 
longer  of  public  benefit.  We  also  proposed  that  he  require 
the  Director  of  NIB  to: 

— Develop  a  system  to  identify  noncompeting  research 
grants  with  less  scientific  merit  than  approved  un¬ 
funded  competing  applications  and  to  assess  which 
grants  should  be  funded. 

— Establish  specific  guidelines  delineating  what  should 
be  incluaed  in  the  annual  scientific  progress  report 
and  instructions  to  grant  administrators  on  how  to 
review  noncompeting  grants.  Written  comments  on  such 
reviews  should  be  incorporated  into  the  official  grant 
files. 


— Clarify  the  specific  authority  of  grant  administrators 
to  withhold  funds  and  disseminate  such  information  to 
all  research  grant  officials. 

--Require  all  institutes  to  use  normalized  priority 
scores  in  funding  research  grants. 

HEW  COMMENTS  AND  OUR  EVALUATION 


HEW  generally  disagreed  with  our  proposals  (see  app.  I) 
and  stated  that  implementing  them 

— woula  require  major  alterations  in  NIH  program  prac¬ 
tices  and  pnilosophies , 

--would  be  extremely  costly,  with  little  prospect  of 
appreciable  benefits,  and 

— might  seriously  damage  the  NIH  extramural  program. 
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we  believe  HEW  can  implement  our  recommendations  in  a  far 
simpler  and  more  effective  manner  than  it  has  indicated, 
with  little  or  no  additional  cost.  We  continue  to  believe 
that  proper  implementation  of  our  recommendations  would  re¬ 
sult  in  more  meritorious  grants  being  funded. 

System  to  identify  noncompeting 
grants  f o r_ termi n at  ion 

HEW  disagreed  with  our  proposal  and  offered  the  follow¬ 
ing  reasons  for  continuing  the  present  methods  for  funding 
grants. 

Priority  scores  are  subjective,  that  is,  not  exact  in¬ 
dicators  of  the  scientific  merits  of  grant  applications. 

Real  differences  among  the  scientific  merits  of  grant  appli¬ 
cations  are  probably  reflected  by  differences  of  30  or  more 
points  in  priority  scores. 

Terminating  grants  early  would  result  in  a  failure  to 
honor  previous  commitments  and  create  an  undesirable  in¬ 
stability  within  the  biomedical  community. 

The  productivity  of  grants  correlates  with  the  original 
review  judgments  of  the  peer  review  groups.  In  addition,  the 
approval  rate  for  competing  renewal  applications  is  signifi¬ 
cantly  higher  than  the  rate  for  new  grant  applications,  thus, 
confirming  that  the  peer  review  groups  screen  out  most  new 
grant  applications  not  likely  to  succeed. 

Host  projects  take  2  to  3  years  to  produce  the  tangiole 
results  needed  to  evaluate  progress,  and  3  years  is  the 
average  duration  of  a  grant  recommended  by  peer  review  groups. 
Grantees  funded  for  3  years  submit  a  renewal  application 
about  1  year  before  the  project  period  expires.  This  makes 
the  first  2  years  of  the  research  available  to  the  peer  re¬ 
view  groups  for  consideration. 

To  monitor,  evaluate,  and  judge  grantee  progress  with  a 
view  toward  termination  would  require  the  quality,  quantity, 
and  variety  of  expertise  now  contained  within  the  peer  review 
system  used  for  assessing  competing  grant  applications.  The 
review  of  ongoing  grants  by  NIH  scientists  could  not  be 
highly  detailed  because  of  their  heavy  workload.  Large  com¬ 
mitments  of  staff  to  evaluate  progress  are  not  justified  be¬ 
cause  not  that  many  projects  would  be  identified  for  termina¬ 
tion. 


We  agree  with  HEW  1 s  comment  that  priority  scores  only 
indicate  scientific  merit  and  suggested  in  the  report  that 
priority  scores  be  used  to  identify  noncompeting  grants  that 
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should  oe  considered  for  ter minat ion .  we  further  suggested 
that  normalized  priority  scores  would  be  particularly  valu¬ 
able  for  this  purpose  because  they  measure  peer  review  judg¬ 
ments  and  can  oe  compared  among  the  various  review  groups. 
After  such  grants  are  identified,  tney  should  be  further 
assessed,  not  terminated  automatically,  before  funds  are 
denied  to  applications  with  greater  scientific  merit. 

In  addition,  because  HEW  believes  that  real  differences 
in  scientific  merit  are  reflected  by  differences  of  30  or 
more  points  in  priority  scores,  it  could  use  the  30  point 
difference  to  identify  those  grants  that  should  be  further 
assessed.  Of  the  126  noncompeting  funaea  grant  applications 
we  identified  as  having  equal  or  worse  priority  scores  than 
competing  applications  not  funded  (see  p.  6),  28  had  priority 
scores  which  differed  by  30  or  more  points  than  some  compet¬ 
ing  grants  not  funded.  Based  upon  HEw's  comments,  the  scien¬ 
tific  merit  differed  among  these  23  grants  and  the  grant  ap¬ 
plications  which  were  not  funded. 

During  the  hearings  before  the  Senate  Committee  on  Ap¬ 
propriations  in  1975,  officials  from  several  dIH  institutes 
stated  that  at  the  proposed  budget  level,  they  would  not  fund 
lower  priority  noncompeting  grants.  According  to  officials 
from  the  two  institutes  we  contacted  regarding  these  state¬ 
ments,  tney  would  first  consider  priority  scores  in  selecting 
noncompeting  grants  for  withholding  action  and  then  consider 
factors  such  as  productivity  and  importance  of  the  projects. 
Those  grants  with  tne  worst  priority  scores  would  probably 
be  the  ones  from  which  funds  would  be  withheld.  Such  use 
of  priority  scores  in  considering  noncompeting  grants  for 
termination  would  be  consistent  with  our  proposal. 

We  agree  with  HEw's  statement  that  cutting  off  funds 
for  noncompeting  grants  would  result  in  failure  to  honor 
previous  commitments,  because  current  regulations  restrict 
HEw's  aoility  to  terminate  grants.  Under  current  regula¬ 
tions,  which  we  proposed  be  changed,  HEW  can  unilaterally 
terminate  a  grant  only  if  the  grantee  has  materially  failed 
to  comply  with  the  terms  and  conditions  of  a  grant.  HEW 
could  change  its  regulations  to  (1)  clearly  require  the 
grantee  to  make  reasonable  progress  as  a  material  condition 
of  grant  performance,  (2)  permit  termination  when  signifi¬ 
cantly  better  applications,  as  evidenced  by  priority  scores, 
could  be  funded,  and  (3)  permit  termination  where  research 
ceased  to  be  of  important  public  benefit.  If  the  regulations 
were  changed  and  future  grantees  were  notified  that  HEW  re¬ 
serves  the  right  to  terminate  grants  in  such  instances,  HEW 
could  redirect  the  remaining  funds  to  better  applications. 


15 


Officials  in  two  other  research  organizations  that  can 
terminate  grants  said  this  authority  had  no  effect  on  the 
quantity  and  quality  of  their  grant  applications  and  that 
their  organizations  receive  many  more  applications  than  can 
be  funded.  The  relatively  few  grants  that  would  be  affected 
by  our  proposals  (using  HEW' s  criteria  of  when  real  differ¬ 
ences  in  scientific  merit  are  reflected  by  priority  scores) 
would  not  appear  to  adversely  affect  the  stability  of  the 
scientific  community. 

HEW's  comments  on  the  value  of  the  peer  review  groups 
for  screening  out  applications  not  likely  to  succeed  and  the 
productivity  of  approved  grants  seem  to  strongly  support  our 
proposal.  For  NIH  not  to  fund  an  application  of  greater 
scientific  merit  in  the  judgment  of  the  peer  review  groups 
in  favor  of  an  ongoing  grant  with  lesser  scientific  merit 
means  that  NIH  is  not  even  considering  funding  those  appli¬ 
cations  with  the  highest  productivity  potential. 

HEW's  comment  concerning  the  time  required  to  evaluate 
the  progress  of  a  grantee  seems  to  refer  to  the  assessment 
that  would  be  required  of  those  grants  with  less  apparent 
scientific  merit  than  the  applications  that  could  not  be 
funded.  NIH  should  use  the  latest  information  available — 
such  as  progress  reports  or  renewal  applications  submitted 
before  the  project  period  expires — to  determine  whether  the 
grants  should  be  terminated.  As  noted  by  HEW,  renewal  ap¬ 
plications,  including  the  status  of  the  first  2  years  of  work, 
are  submitted  to  NIH  about  1  year  before  the  project  period 
expires.  The  peer  review  groups  use  this  data  to  approve  or 
disapprove  the  application  and  to  award  it  a  priority  score. 
Such  evaluations  do  not  affect  the  project  period  of  the 
existing  grant,  just  the  new  grant  being  applied  for.  Such 
information  could  be  used  to  assess  whether  the  current  grant 
should  be  terminated. 

Other  information  to  be  considered  in  terminating  a 
grant  is  the  amount  of  funds  not  yet  obligated  by  the 
grantee  which  could  become  available  to  NIH  for  funding 
applications  with  significantly  greater  scientific  merit. 

If  NIH  would  prefer  to  have  the  advice  of  the  peer  re¬ 
view  groups  in  determining  whether  to  terminate  grants  with 
less  scientific  merit  than  unfunded  applications,  we  would 
agree  with  them.  We  ao  not  believe  our  proposal  would  re¬ 
quire  a  large  increase  in  staff,  as  stated  by  HEW,  if  prior¬ 
ity  scores  were  used  to  screen  grants  for  further  assessment. 
If  HEW  assessed  those  ongoing  grants  with  priority  scores  of 
30  points  or  more  than  competing  applications  that  could  not 
be  funded,  only  28  grants  in  the  institutes  reviewed  would 
have  had  to  be  studied  further  in  fiscal  year  1974.  This 
would  not  appear  to  place  an  unreasonable  burden  on  NIH 
grant  administrators  or  the  peer  review  groups. 


16 


Proposal  to  establish  guidelines  for 
preparation  and  review  of  progress _ reports 

Since  grantees  might  not  have  guidelines  for  annual  re¬ 
porting,  HEW  agreed  to  have  NIH  survey  the  use  of  structured 
progress  reports  and,  if  necessary,  develop  instructions  for 
grantees  to  use  in  preparing  such  reports.  HEW  also  agreed 
that  NIH  staff  lack  formalized  NIH-wide  instructions  for 
reviewing  progress  reports  and  taking  appropriate  actions 
on  problems.  HEW  agreed  to  have  NIH  develop  appropriate 
NIH-wide  instructions. 

HEW  stated  that  because  of  limited  staff  NIH  cannot 
determine  the  merit  of  scientific  progress  each  year  for 
14,000  projects  by  reviewing  annual  progress  reports.  It 
can  only  be  assured  that  the  research  continues  within  the 
scope  of  the  original  project  and  that  the  investigator  has 
shared  with  NIH  the  findings  made  during  the  previous  year. 

HEW  saw  no  need  for,  or  appreciable  benefits  to  be 
gained  from,  the  scientific  staff  preparing  written  comments 
on  their  review  of  progress  reports  unless  they  determined 
that  problems  exist.  HEW  further  stated  that  while  program 
staff  already  document  problems  they  have  identified  and  re¬ 
sulting  actions,  it  will  require  that  grant  files  be  appro¬ 
priately  documented  when  problems  are  encountered.  Most 
scientist  administrators  we  contacted  said  they  handle  such 
problems  orally.  Therefore,  the  need  for  NIH  to  require  that 
grant  files  be  appropriately  documented  is  more  significant 
than  HEW  indicated. 

We  have  not  reviewed  the  adequacy  of  NIH  staffing  but 
we  believe  NIH  must  assure  itself  that  the  14,000  projects 
involving  millions  of  dollars  annually  are  making  satisfac¬ 
tory  progress  toward  the  goals  specified  in  the  grant  appli¬ 
cations.  To  alleviate  the  workload,  NIH  could  develop  a 
simple  form  requiring  the  grant  administrator  to  indicate 
whether  the  grantee's  progress  is  excellent,  satisfactory, 
fair,  or  poor  and  to  provide  a  short  notation  explaining  the 
position.  This  would  help  NIH  officials  compare  the  per¬ 
formance  of  grantees  and  could  be  used  with  other  indicators 
for  determining  whether  the  grant  should  be  terminated. 

Proposal  to  clarify  authority  of 
grant  administrators  to  withhold  funds 

HEW  agreed  with  our  proposal  and  said  the  Division  of 
Grants  and  Contracts,  Office  of  the  Assistant  Secretary  for 
Health,  has  begun  developing  guidelines  in  this  area  for  use 
throughout  the  Public  Health  Service. 
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Proposal _ to  change  regulations 

for  terminating  grants 
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HEW  did  not  agree  with  our  proposal  to  permit  selec- 
termination  of  grants  in  cases  where  (1)  there  is  poor 
ress,  (2)  better  applications  could  be  funded,  or 
the  research  is  no  longer  of  public  benefit.  It  stated 
existing  regulations  and  policies  permit  the  grantor  to 
aterally  terminate  the  support  of  a  project  when  a  grantee 
s  to  show  satisfactory  progress  and  for  other  reasons, 
uding  a  determination  that  the  research  is  no  longer  of 
ic  benefit.  Since  NIH  must  approve  at  the  end  of  the 
the  funds  a  grantee  carries  over,  HEW  said  NIH  can  pre- 
a  grantee  from  carrying  over  unexpended  funds.  HEW  did 
specify  under  what  conditions  such  a  mechanism  can  be 


HEW  stated  that  grants  do  not  often  need  to  be  termi¬ 
nated.  Because  of  the  way  grants  are  approved  and  funded, 

HEW  said  that  grants  of  poor  quality  or  lacking  relevance  to 
important  health  problems  are  seldom  encountered. 

We  do  not  agree  with  HEW 1 s  statement  that  existing 
policies  and  regulations  provide  the  authority  for  NIH  to 
unilaterally  terminate  a  grant  because  the  research  is  no 
longer  of  public  benefit.  They  do  not,  for  example,  allow 
NIH  to  terminate  when  recent  scientific  developments  or 
other  events  result  in  research  being  no  longer  of  public 
benefit.  Further,  they  do  not  specifically  provide  that 
NIH  can  unilaterally  terminate  a  grant  when  progress  is  poor. 
The  sources  HEW  cited  show  that  NIH  can  withhold  funds  under 
such  conditions  but  that  a  grant  can  be  terminated  only  if 
the  grantee  has  materially  failed  to  comply  with  the  terms 
and  conditions  of  the  grant.  We  noted  that  reasonable  prog¬ 
ress  is  not  routinely  made  a  material  term  and  condition  of 
the  grant  award.  We  believe  HEW  should  revise  its  regula¬ 
tions  to  clearly  state  the  conditions  under  which  it  can 
terminate  a  grant. 

We  do  not  believe  that  NIH  can  prevent  a  grantee  from 
spending  funds  already  awarded.  As  stated  on  page  9,  an 
opinion  by  HEW ' s  legal  counsel  stated  that  a  grantee  can  ex¬ 
pend  funds  already  granted  unless  the  grantee  has  done  some¬ 
thing  wrong.  An  NIH  official  responsible  for  policies  and 
procedures  for  research  grants  said  that,  although  NIH  may 
choose  not  to  award  future  funds  under  a  research  grant,  NIH 
cannot  reduce  the  approved  project  period  during  which  pre¬ 
viously  awarded  funds  can  be  obligated  by  the  grantee.  We 
believe  that  termination  authority  is  the  only  means  avail¬ 
able  for  NIH  to  prevent  a  grantee  from  obligating  grant  funds 
that  were  awarded  during  the  budget  year. 
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HEW's  comment  that  grants  of  poor  quality  or  lacking 
relevance  to  important  health  problems  are  seldom  encountered 
may  very  well  be  true.  Once  such  grants  are  encountered, 
however,  NIH  presently  cannot  terminate  them.  In  addition, 
the  term  "poor  quality"  is  relative.  As  shown  on  page  6, 

126  funded  grants  were  of  equal  or  poorer  quality,  based  on 
priority  scores,  than  applications 
118  of  the  126  had  priority  scores 
and  26  were  in  the  category  which, 
scientific  differences. 


that  were  not  funded; 
indicating  poorer  quality 
according  to  HEW,  had  real 


Proposal  to  require  use  of  normalized  scores 

HEW  agreed  to  explore  further  the  use  of  normalized 
priority  scores  in  funding  research  grants.  HEW  stated 
that,  although  the  normalization  process  has  been  extensively 
studied  in  the  past,  the  NIH  Grants  Peer  Review  Study  Team, 
recently  established  by  the  Director,  NIH,  to  examine  all 
aspects  of  the  NIH  peer  review  system,  would  further  evaluate 
use  of  normalized  priority  scores. 


We  believe  a  numerical  ranking  system  is  necessary  as  an 
aid  in  selecting  grants  for  funding  and  that  ranking  by  raw 
priority  scores  is  not  acceptable,  because  they  are  awarded 
by  more  than  one  study  section.  We  believe  NIH  should  give 
priority  to  studying  its  process  of  normalizing  priority 
scores  so  that  one  acceptable  NIH-wide  system  for  ranking 
grant  applications  for  funding  can  be  implemented. 

RECOMMENDATIONS 

We  recommend  that  the  Secretary  of  HEW  change  the 
regulations  for  terminating  research  grants  to  clearly  re¬ 
quire  the  grantee  to  make  reasonable  progress  as  a  material 
condition  of  grant  performance  and  to  permit  selective 
termination  of  future  grants  when  (1)  significantly  better 
applications,  as  evidenced  by  priority  scores,  could  be 
funded  or  (2)  recent  scientific  developments  or  other  events 
result  in  research  being  no  longer  of  public  benefit.  We 
also  recommend  that  he  require  the  Director,  NIH,  to; 

— Develop  a  system  to  identify  noncompeting  research 
grants  with  significantly  less  scientific  merit  than 
approved  unfunded  competing  applications  and  to  assess 
which  grants  should  be  funded.  The  30-point  differ¬ 
ence  in  priority  scores,  which  HEW  considers  to  re¬ 
flect  real  differences  in  scientific  merit,  could  be 
used  to  identify  those  grants  to  be  further  assessed. 
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— Establish  specific  guidelines  delineating  what  should 
be  included  in  the  annual  scientific  progress  report 
and  instructions  to  grant  administrators  on  how  to 
revise  noncompeting  grants.  Written  comments  on  such 
reviews  should  be  incorporated  into  the  official  grant 
files. 

— Clarify  the  specific  authority  of  grant  administrators 
to  withhold  funds  and  disseminate  such  information  to 
all  research  grant  officials. 

— Give  priority  to  studying  its  process  of  determining 
normalized  priority  scores  and  require  all  institutes 
to  use  it. 
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CHAPTER  3 


TERMINAL  REPORTS  NOT  BEING  RECEIVED 

Each  research  grant  the  National  Institutes  of  Health 
awards  is  made  subject  to  the  condition  that  the  grantee 
maintain  technical  progress  and  fiscal  reports  and  submit 
them  to  the  awarding  institute  on  a  predetermined  basis. 

In  addition,  grantees  must  certify  to  NIH  whether  or  not 
inventions  have  been  made  as  a  result  of  a  grant  award. 

NIH  requires  such  reports  annually  and  at  the  end  of  the 
total  project  period  for  ongoing  grants.  The  reports  sub¬ 
mitted  at  the  end  of  a  project  period  are  referred  to  as 
terminal  reports.  At  the  institutes  reviewed,  terminal 
reports  were  not  being  received,  in  part  because  the  in¬ 
stitutes  were  not  closely  monitoring  the  submission  of  such 
reports . 

Although  NIH  requires  the  institutes  to  contact  the 
grantee  to  obtain  delinquent  reports,  in  some  cases  the 
institutes  were  not  following  this  policy  and  in  other  cases 
grantees  had  been  notified  but  still  did  not  submit  a  delin¬ 
quent  report.  NIH  continues  to  support  researchers,  even  if 
they  have  not  complied  with  the  conditions  of  prior  grant 
awards.  NIH  policy  does  not  prohibit  the  funding  of  grantees 
who  have  not  filed  delinquent  reports  required  under  other 
NIH  support. 

PROBLEMS  IN  OBTAINING  REQUIRED  REPORTS 

The  terminal  progress  report  must  include  a  summary 
statement  of  progress  made  toward  achieving  the  originally 
stated  aims  of  the  grant  proposal,  a  list  of  the  significant 
results,  and  a  list  of  publications  resulting  from  the 
research.  According  to  NIH,  the  progress  report  is  considered 
a  valuable  guide  for  the  institutes  in  evaluating  the  work 
accomplished  under  the  grant.  If  the  terminal  progress  report 
is  not  submitted  within  90  days  after  the  termination  of  a 
grant,  the  report  is  considered  delinquent.  After  120  days, 
the  respective  institute  must  notify  the  grantee  of  that 
delinquency . 

The  November  1971  NIH  guideline,  which  addresses  terminal 
reports,  states:  "A  recurring  problem  in  the  administration 
of  many  NIH  grant  programs  is  the  delinquency  on  the  part  of 
some  grantees  in  submitting  reports  required  as  a  condition 
of  the  grant  award."  The  same  document  directs  the  institutes 
to  insure  the  submission  of  such  reports. 
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At  the  end  of  calendar  year  1974,  grantees  of  at  least 
321  research  grants  were  delinquent  in  submitting  required 
reports  to  National  Institute  of  Child  Health  and  Human  De¬ 
velopment  and  National  Institute  of  Allergy  and  Infections 
Diseases.  In  NICHD ,  164  grants  that  had  terminated  between 
fiscal  year  1969  and  fiscal  year  1974  had  missing  terminal 
reports.  In  NIAID,  for  157  of  402  grants  that  had  terminated 
in  fiscal  years  1972  and  1973,  the  grantees  had  not  submitted 
all  the  required  terminal  reports.  At  the  time  of  our  re¬ 
view,  NIAID  officials  did  not  know  for  how  many  of  the  1974- 
terminated  grants  the  grantees  were  delinquent  in  submitting 
reports.  In  addition,  a  National  Institute  of  Environmental 
Health  Sciences  official  said  the  institute  did  not  have  a 
system  to  insure  that  terminal  reports  were  submitted  as 
required.  Grantees  were  delinquent  in  submitting  terminal 
progress  reports  for  all  21  research  grants  that  terminated 
in  fiscal  year  1973  and  the  first  half  of  fiscal  year  1974. 


Although  many  grantees  of  NICHD  and  NIAID  submitted 
required  reports  when  notified  by  the  institutes,  several 
were  notified  repeatedly  and  still  did  not  submit  delinquent 
reports.  In  many  cases,  these  reports  were  over  2  years 
delinquent.  At  the  time  of  our  review,  the  terminal  progress 
report  in  both  institutes  was  the  one  document  most  often 
delinquent.  For  example,  of  40  NICHD  research  grants  that 
terminated  in  fiscal  year  1973,  in  late  1974,  80  percent  of 
the  terminal  progress  reports  were  missing  while  70  percent 
of  the  invention  statements  were  absent  from  the  files  re¬ 
viewed.  Of  a  sample  of  28  NIAID  grants  that  terminated 
between  June  1971  and  December  1972,  20  grantees  were  delin¬ 
quent  in  submitting  the  terminal  progress  report.  As  of 
November  1974,  17  grantees  still  had  not  submitted  such 
reports . 


TERMINAL  PROGRESS  REPORTS 
NOT  ALWAYS  COMPREHENSIVE 


Some  terminal  progress  reports  submitted  by  grantees 
and  accepted  by  the  institutes  did  not  contain  information 
on  all  research  conducted  during  the  entire  grant  period. 

In  some  cases,  an  entire  year's  research  results  were  not 
reported . 

If  a  grantee  re-applies  for  NIH  support  and  the  applica¬ 
tion  is  disapproved,  the  grant  expires.  In  these  cases,  NICHD 
and  NIAID  accepted  the  progress  report,  which  accompanied 
the  renewal  application,  as  the  terminal  progress  report, 
although  this  report  did  not  cover  the  entire  grant  period. 
Although  we  were  told  this  practice  conformed  with  NIH  policy, 
no  written  guidelines  or  policy  statements  sanctioned  the 
practice . 
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According  to  NIH  instructions,  the  competing  renewal 
application,  along  with  a  progress  report,  must  be  submitted 
to  NIH  for  consideration  8  months  before  the  original  grant 
expires  for  funding  not  to  be  interrupted.  As  discussed 
previously,  the  three  institutes  did  not  fund  approximately 
44  percent  of  competing  renewal  research  grant  applications. 
This  rate  is  similar  to  NIH's  overall  rate.  The  progress 
reports  accepted  for  these  competing  renewal  research  grant 
applications  that  are  not  funded  sometimes  do  not  report  on 
research  accomplishments  for  almost  the  entire  last  year  of 
NIH  support.  However,  grant  administrators  in  all  three 
institutes  said  researchers  normally  use  the  last  year  of  a 
research  grant  to  integrate  findings  and  arrive  at  conclusions. 
Accordingly,  they  believed  the  last  year  of  a  research  grant 
was  the  most  important  from  NIH's  standpoint. 

To  determine  the  extent  that  research  findings  were  not 
fully  reported,  we  sampled  17  terminated  research  grants  for 
which  NICHD  and  NIAID  accepted  terminal  progress  reports. 
Institute  personnel  accepted  11  of  these  reports  before  the 
end  of  the  original  grant  period,  although  they  had  been 
submitted  with  a  competing  renewal  application.  These 
grantees  had  not  reported  the  results  of  the  research  for 
the  entire  project  period.  For  example,  one  NICHD  research 
grant  ended  in  May  1969,  but  the  terminal  progress  report 
was  submitted  in  April  1968  with  the  competing  renewal  appli¬ 
cation.  Thus,  research  results  for  13  months  were  not 
reported.  In  another  case,  grant  support  ended  in  May  1973, 
and  the  progress  report  was  submitted  and  accepted  in  August 
1972  . 


We  asked  institute  personnel  why  this  practice  was  per¬ 
mitted.  They  replied  that  (1)  they  rely  on  the  grantee  to 
publish  research  results  and,  therefore,  make  the  findings 
known  and  (2)  grant  administrators'  workloads  make  it 
impossible  to  follow  up  on  every  research  grant  to  obtain 
more  complete  terminal  progress  reports. 

RESEARCHERS  IN  VIOLATION 
OF  ONE  GRANT  FUNDED  BY  ANOTHER 

NIH  has  no  formal  system  of  communicating  information 
within  and  among  institutes  about  grantees  who  have  not 
complied  with  the  terms  of  a  research  grant  by  not  submitting 
required  reports.  As  a  result,  grantees  who  had  not  sub¬ 
mitted  terminal  reports  under  one  research  grant  were  being 
supported  by  NIH  under  another  grant  or  contract. 

NIH  requires  that  all  grantee  requests  for  continuing 
a  noncompeting  research  grant  be  accompanied  by  interim 
progress  and  financial  reports  as  a  prerequisite  for  continued 
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support.  Lacking  such  reports,  funding  for  ongoing  research 
is  withheld.  However,  reporting  requirements  under  a  completed 
grant  are  not  required  to  be  fulfilled  as  a  prerequisite  of 
support  under  another  grant.  NIH  regulations  only  state 
that,  if  a  grantee  continues  to  be  delinquent  in  submitting 
a  terminal  progress  report,  the  awarding  institute  may  notify 
the  grantee  that  it  will  not  fund  additional  grants  in  which 
the  researcher  is  involved  until  the  report  is  received.  How¬ 
ever,  these  regulations  do  not  prohibit  a  grantee  that  is 
delinquent  in  submitting  terminal  reports  from  obtaining 
another  grant. 

We  randomly  selected  15  researchers  with  expired  NICHD 
grants  for  which  terminal  reports  had  not  been  submitted 
to  determine  if  NIH  was  currently  supporting  them.  Seven 
were  being  funded  under  research  grants  or  contracts  by 
either  NICHD  or  other  institutes,  although  the  researcher 
had  not  complied  with  the  terms  of  a  previous  grant.  Only 
one  of  the  seven  researchers  submitted  a  terminal  progress 
report,  but  even  that  researcher  failed  to  submit  an  invention 
statement.  Another  researcher  did  not  submit  an  invention 
statement  but  did  submit  a  progress  report  for  part  of  the 
grant  period  when  an  unfunded  competing  renewal  application 
was  submitted.  The  remaining  researchers  did  not  submit 
competing  renewal  applications  or  terminal  progress  reports. 

In  fiscal  year  1974,  these  seven  researchers  were  partici¬ 
pating  in  eight  different  projects  supported  by  NIH.  These 
researchers '  terminal  reports  from  previous  grants  were 
delinquent  anywhere  from  3  to  80  months.  NICHD  was  again 
funding  six  of  the  seven  researchers.  Total  support  during 
fiscal  year  1974  for  these  eight  projects  amounted  to  at 
least  $1.5  million.  One  researcher  was  being  supported  by 
NICHD  under  a  research  grant  for  about  $105,000,  although  he 
had  not  submitted  reports  on  two  previous  grants  that  NICHD 
had  also  funded. 

NICHD  personnel  said  no  established  procedure  existed 
where  one  institute  would  routinely  be  informed  of  researchers, 
supported  by  another  institute,  who  were  delinquent  in  filing 
terminal  reports.  In  addition,  these  personnel  have  not  been 
requested,  nor  do  they  systematically  inform  institute  offi¬ 
cials,  of  researchers  who  have  not  complied  with  the  terms 
of  a  previous  grant  award. 

CONCLUSIONS 

At  all  the  institutes  reviewed,  terminal  progress  reports 
and  final  invention  statements  were  not  being  submitted  as 
required  under  the  conditions  of  the  grant  award.  In  addition, 
one  institute  had  no  way  to  readily  identify  which  grants 


had  delinquent  reports  or  which  researchers  were  responsible 
for  those  reports.  As  a  result,  the  research  results  accruing 
under  some  grants  were  not  being  reported  to  NIH. 

NIH  policy  does  not  prohibit  its  institutes  from  funding 
researchers  who  are  delinquent  in  submitting  reports.  Conse¬ 
quently,  NIH  is  supporting  some  researchers  even  though  they 
are  violating  the  conditions  of  previous  research  grant  awards. 
Neither  the  institutes  reviewed  nor  NIH  has  a  system  for 
coordinating  and  exchanging  information  on  researchers  responsi¬ 
ble  for  delinquent  reports.  Therefore,  not  funding  a  particu¬ 
lar  researcher  would  be  difficult,  even  if  NIH  desired  to  do 
so . 


Because  the  terminal  progress  report  is  the  only  document 
required  under  the  grant  that  summarizes  a  grantee's  research 
for  the  full  grant  period,  we  believe  NIH  should  establish  a 
system  for  obtaining  the  reports  at  the  conclusion  of  the 
grant.  Future  funds  should  be  denied  a  grantee  if  the  termi¬ 
nal  reports  are  not  submitted. 

RECOMMENDATIONS 

The  Secretary  of  the  Department  of  Health,  Education, 
and  Welfare  should  instruct  the  Director,  NIH,  to: 

--Require  all  institutes  to  more  closely  monitor  the 
submission  of  required  reports  when  a  grant  expires. 

--Prohibit  the  acceptance  of  terminal  progress 
reports  that  do  not  cover  the  entire  period  of 
grant  support. 

--Prohibit  the  funding  of  researchers  when  they  are 
known  to  be  violating  the  terms  and  conditions  of 
previous  NIH  support. 

--Establish  an  information  system  capable  of  ex¬ 
changing  information  on  delinquent  research  grant 
reports  among  institutes. 

HEW  COMMENTS  AND  OUR  EVALUATION 

HEW  agreed  that  terminal  progress  reports  should  be 
sumitted  and  that  they  should  cover  the  entire  grant  period. 
HEW  did  not  state  what  steps  it  plans  to  take  to  implement 
our  recommendations.  In  addition,  HEW  did  not  respond  to 
our  recommendations  concerning  (1)  not  funding  researchers 
who  have  violated  the  conditions  of  previous  support  by  not 
submitting  terminal  reports  and  (2)  the  need  for  a  system 
for  exchanging  information  on  such  researchers  among  in¬ 
stitutes  . 
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Since  we  do  not  know  how  HEW  plans  to 
pronibiting  funding  of  researchers  who  hav 
terminal  progress  reports  covering  the  ent 
grant  support  and  establishing  a  system  fo 
information  on  such  researchers  would  be  e 
to  get  all  required  reports  from  grantees. 


address  the  problem, 
e  not  submitted 
ire  period  of 
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CHAPTER  4 


INTRAMURAL  RESEARCH 

National  Institutes  of  Health  scientists  conduct 
intramural  research  in  their  own  laboratories.  The  in¬ 
tramural  research  programs,  based  on  a  branch  system,  are 
headed  by  a  scientific  director  or  comparable  official  of 
the  institute  who  controls  and  directs  the  branches  by 
allocating  space,  personnel,  positions,  and  funds  to  each 
branch.  The  head  of  each  branch  reports  to  the  scientific 
director  of  the  institute.  One  or  more  sections,  each  de¬ 
voted  to  specific  research  areas  within  a  scientific  dis¬ 
cipline,  may  be  within  each  branch. 

NIH  has  no  formal  procedures  for  initiating  most  intra¬ 
mural  research  projects.  A  number  of  ongoing  projects  were 
terminated  because  an  independent  review  of  the  projects 
showed  they  were  either  not  related  to  the  institute's 
mission  or  their  scientific  merit  was  questionable. 

LACK  OF  FORMAL  PROCEDURES  FOR 
INITIATING  INTRAMURAL  RESEARCH 

Except  for  clinical  investigations  of  human  subjects 
and  research  involving  hazardous  materials,  NIH  has  no  formal 
policies  or  procedures  for  initiating  new  intramural  labora¬ 
tory  research  projects.  Research  projects  are  not  subject  to 
peer  review  for  scientific  merit,  as  required  in  the  extra¬ 
mural  grant  programs,  and  the  policies  and  procedures  used 
to  start  new  projects  are,  for  the  most  part,  informal  and 
are  established  at  the  discretion  of  each  branch  chief. 

New  research  projects  in  intramural  research  programs 
are  generally  reviewed  through  discussions  between  the 
researchers  or  principal  investigator  of  a  proposed  project 
and  the  section  or  branch  chief.  If  the  principal  investi¬ 
gator  is  a  branch  chief,  he  generally  approves  his  own 
projects.  Although  branch  chiefs  discuss  the  initiation  of 
research  projects  with  the  institutes'  scientific  directors, 
in  some  cases  this  is  not  required  and  projects  generally 
receive  no  higher  approval  than  that  of  the  respective 
branch  chiefs.  None  of  the  institutes  reviewed  required 
written  justifications  or  research  plans. 

Although  a  few  branch  chiefs  had  independently  estab¬ 
lished  formal  controls  over  the  initiation  of  research  within 
their  own  branches,  most  branches  did  not  require  new  proj¬ 
ects  to  be  subject  to  peer  review  before  they  were  begun. 
However,  some  branch  chiefs  said  a  branch  chief  might  not 
have  the  expertise  necessary  to  assess  the  scientific  merit 
of  all  projects  planned  by  the  researchers  they  supervise. 
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According  to  branch  chiefs  who  had  instituted  formal 
procedures,  written  research  plans  force  a  researcher  to 
articulate  research  goals  and  methods  to  be  used  and  provide 
documentation  upon  which  assessments  can  be  made.  One  branch 
chief  said  formal  research  plans  and  peer  review  of  proposals 
were  essential  to  good  manaqement. 

In  the  National  Institute  of  Child  Health  and  Human 
Development,  one  branch  requires  a  researcher  to  submit  a 
written  plan  before  a  new  research  project  is  initiated.  The 
plan  is  reviewed  and  criticized  by  the  researcher's  peers 
within  the  branch.  If  this  peer  group  disapproves  of  the 
proposed  project,  it  will  not  be  funded.  The  chief  of  this 
branch  said  the  peer  review  group  annually  disapproves  about 
six  projects  after  receiving  his  tentative  approval. 

In  the  3  institutes  reviewed,  only  3  of  the  19  branch 
chiefs  required  written  plans  before  the  laboratory 
research  project  was  initiated.  Thirteen  branch  chiefs 
discussed  projects  with  researchers  before  they  were  initiated, 
while  two  others  said  no  approval  was  required.  One  branch 
chief  required  written  plans  if  the  resources  to  be 
committed  were,  in  his  opinion,  significant.  Of  50  research 
projects  started  in  fiscal  year  1974  at  the  National  In¬ 
stitute  of  Environmental  Health  Sciences,  18  were  approved 
after  written  research  plans  were  submitted  to  branch 
chiefs,  and  32  were  informally  approved  after  discussion 
with  the  research  scientist. 

Most  branch  chiefs  are  engaged  in  research  and,  there¬ 
fore,  can  approve  their  own  research  projects.  During  fiscal 
year  1974,  10  branch  chiefs  in  NICHD  and  NIEHS  were  personally 
involved  in  23  research  projects.  Eleven  NIEHS  projects 
alone  cost  about  $528,000  in  fiscal  year  1974. 

The  board  of  scientific  counselors  within  each  institute 
is  appointed  by  the  Director,  NIH,  under  the  authority  of 
the  Public  Health  Service  Act.  The  board  consists  of  leaders 
in  basic  and  clinical  research,  and  its  function  is  to  ad¬ 
vise  the  institute  director  on  scientific  matters  relating  to 
intramural  research  programs.  However,  in  none  of  the 
institutes  reviewed  were  these  boards  reviewing  intramural 
research  projects  before  they  were  initiated.  One  institute 
director  said  the  board  does  not  review  projects  before  they 
are  initiated,  because  he  believes  scientists  should  have  a 
minimum  of  outside  intervention.  Another  official  felt 
scientific  freedom  was  necessary  for  researchers  to  function. 
We  were  also  told  that  the  limited  number  of  board  members 
made  it  difficult  to  insure  the  expertise  necessary  to  review 
all  intramural  research  projects. 
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The  institutes'  current  methods  of  initiating  intramural 
research  projects  could  be  improved  to  better  insure  the 
allocation  of  resources  to  the  highest  quality  of  research 
otherwise  attainable.  For  example,  NIEHS  officials  termi¬ 
nated  14  projects  during  fiscal  years  1973  and  1974;  11  for 
reasons  of  low  research  priority  considering  available 
resources.  These  11  projects  lasted  from  2  to  6  years  and 
cost  about  $1.1  million  before  they  were  terminated.  At  the 
same  institute,  the  board  of  scientific  counselors  reviewed 
the  work  of  32  scientists  during  fiscal  year  1974  and  found 
that  5  were  doing  research  (1)  not  directly  related  to  the 
institute's  mission  or  (2)  of  questionable  scientific  merit. 
None  of  the  projects  terminated  had  been  submitted  to  peer 
review  and  formally  approved  on  the  basis  of  written  research 
plans  before  they  were  started. 

Although  the  board  of  scientific  counselors  at  NICHD 
generally  praised  the  research  efforts  reviewed  during  fiscal 
year  1974,  they  criticized  the  work  of  some  scientists  as 
being  of  questionable  scientific  merit.  The  scientific 
directors  of  both  institutes  generally  agreed  with  the 
opinions  of  the  boards  and  followed  their  recommendations 
by  terminating  the  projects  in  question  or  taking  other 
corrective  action  such  as  redirecting  the  research. 

MONITORING  AND  EVALUATING  ONGOING  RESEARCH 
NEEDS  IMPROVEMENT 


Ongoing  intramural  research  projects  were  monitored  and 
evaluated  through  various  means,  some  of  which  were  common 
to  each  institute. 

The  institutes  usually  monitor  intramural  research 
projects  informally  with,  for  example,  daily  contacts  and 
discussions  between  section  and  branch  chiefs  and  individual 
scientists.  The  institutes  were  also  using  research  seminars, 
involving  intramural  scientists,  at  which  researchers  pre¬ 
sented  their  work  to  their  peers  for  review  and  discussion. 
However,  the  institutes  had  no  formal  requirement  that  these 
seminars  be  held  nor  any  prescribed  format  detailing  which 
researchers  would  present  particular  research  projects  or 
how  often.  The  seminars  operated  largely  at  the  discretion 
of  branch  chiefs. 

Research  programs  are  formally  reviewed  by  the  board  of 
scientific  counselors  at  each  institute.  Officials  at  NICHD 
and  NIEHS  told  us  that  the  boards,  which  meet  semiannually, 
review  all  intramural  research  programs  within  a  particular 
institute  every  2  years.  The  National  Institute  of  Allergy 
and  Infectious  Diseases  board  of  scientific  counselors  reviews 
all  programs  within  a  4-1/2-year  period.  Only  two  boards-- 
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NIEHS  and  NICHD--however ,  were  reviewing  individual  research 
projects  or  the  work  of  individual  scientists.  The  other 
board  was  reviewing  overall  program  direction  of  the  in¬ 
tramural  programs. 

The  NIAID  board  of  scientific  counselors  was  not  review¬ 
ing  ongoing  research  in  detail  but  in  1974  recommended  that 
a  peer  review  group  for  intramural  research  programs  be 
established  to  evaluate  the  scientific  merit  of  work  done  by 
intramural  researchers.  A  November  12,  1974,  NIAID  document 
states  that  "the  Board  of  Scientific  Counselors  cannot  be 
regarded  as  a  peer  review  group  and  that  there  are  inadequa¬ 
cies  in  the  present  NIH  system  to  deal  with  the  problem  of 
allocation  of  resources  *  *  As  a  result,  an  NIAID 

official  said  that  beginning  in  June  1976,  consultants 
will  be  obtained  on  an  ad  hoc  basis  to  help  the  board  of 
scientific  counselors  make  peer  reviews  of  individual 
scientists'  work.  The  consultants  will  be  chosen  for  their 
special  knowledge  in  the  fields  to  be  reviewed. 

As  shown  in  the  preceding  section,  several  projects  were 
found  to  be  of  low  scientific  merit  or  not  related  to  the 
institute's  mission,  after  the  NICHD  or  NIEHS  boards  reviewed 
specific  research  projects. 

CONCLUSIONS 

NIEHS,  NICHD,  and  NIAID  spent  approximately  $43.2  million 
for  intramural  research  in  fiscal  year  1974.  Most  research 
projects  were  initiated  without  formal  peer  review  and  ap¬ 
proval.  In  addition,  some  projects  were  approved  by  the 
same  scientist  who  did  the  research. 

Reviews  of  ongoing  research  by  the  boards  of  scientific 
counselors  indicate  that  NICHD  and  NIEHS  have  funded  research 
of  questionable  scientific  merit  and  relevance  to  the  insti¬ 
tute's  mission.  At  NIAID,  reviews  of  individual  research 
projects  were  not  being  made  at  the  time  of  our  review. 

In  our  opinion,  peer  review  of  intramural  research  proj¬ 
ects  before  they  are  initiated,  as  well  as  when  they  are 
active,  is  essential  to  insuring  that  NIH's  limited  resources 
are  most  effectively  and  efficiently  used.  Although  informal 
mechanisms  being  used  appeared  to  be  beneficial,  such  systems 
have  not  been  successful  in  making  best  use  of  research  funds. 

We  proposed  that  the  Secretary  ot  tne  Department  of 
Health,  Education,  and  Welfare  instruct  the  Director,  NIH, 
to  require  that  written  plans  for  all  intramural  research 
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projects  be  reviewed  and  approved  by  a  peer  review  group 
before  research  projects  are  initiated  and  that  reviews  be 
made  of  specific  ongoing  research  projects.  If  augmented 
by  ad  hoc  consultants,  the  boards  of  scientific  counselors 
could  do  this. 

HEW  COMMENTS  AND  OUR.. EVALUATION 

HEW  did  not  agree  with  our  proposals  and  stated  that 
the  only  justification  for  changing  the  management  of  the 
intramural  research  program  would  be  that  it  was  not 
successful.  HEW  stated  that,  by  any  measurement,  the  in¬ 
tramural  program  has  produced  extremely  well.  HEW  stated 
that  the  present  system  of  management  is  better  because  ic 
includes  (1)  a  continuing  review  of  ongoing  projects,  (2) 
carefully  selecting  people  for  employment,  promotion,  and 
conversion  to  permanent  status,  thus  assuring  high  quality 
people,  and  (3)  a  decisionmaking  process  concerning  the 
quality  and  priority  of  research  each  time  equipment  is  pur¬ 
chased  or  a  technician  is  assigned. 

HEW  also  stated  that  an  advantage  of  the  intramural 
research  program  is  that  it  provides  scientists  with  the 
freedom  to  pursue  long-term  research  without  submitting 
applications.  If  this  advantage  were  taken  away,  many  top 
scientists  would  be  lost  to  other  organizations  offering 
better  benefits. 

HEW  stated  that  the  cost  of  an  outside  peer  review 
system,  plus  its  destructive  side  effects,  would  far  outweigh 
the  good  to  be  achieved  by  weeding  out  a  few  lower  priority 
projects.  A  peer  review  system  would  not  be  able  to  respond 
quickly  to  scientists  awaiting  approval  of  their  work. ^  HEW 
also  stated  that  the  projects  terminated  early  do  not  indicace 
a  weakness  in  the  system  but  a  strength. 

We  did  not  intend  to  suggest  in  our  recommendations  that 

--the  quality  and  productivity  of  intramural  research 
is  lower  than  research  conducted  through  the  NIH  ex¬ 
tramural  program  or  at  any  institution  other  than 
NIH  that  conducts  biomedical  research, 

--NIH  abandon  the  management  practices  it  now  uses,  or 

--NIH  had  to  use  outside  groups  to  approve  a  project 
before  it  was  started  (this  was  mentioned  as  one 
possible  approach). 
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We  did  intend  that  management  of  the  intramural  program 
be  improved  by  requiring  scientists  to  submit  written  plans 
for  approval  before  projects  started  to  avoid  research  in 
an  area  that  should  not  be  performed  or  could  be  better 
designed.  If  NIH  wanted  to  use  intramural  scientists  to 
review  the  research  plans  of  their  peers,  the  results  would 
benefit  both  the  researcher  and  management.  For  example, 
one  branch  chief  requires  written  plans  from  researchers 
and  that  researchers'  peers  within  the  branch  review  and 
criticize  the  plans.  The  branch  chief  stated  that,  as  a 
result,  some  projects  were  disapproved  by  the  peers  after 
he  had  given  tentative  approval.  We  believe  that  such  a 
system  has  merit. 

Why  would  top  scientists  leave  NIH  if  they  were  required 
to  document  their  research  plans  before  they  initiate  the 
research?  They  would  be  exposed  to  expert  advice  from  their 
peers  and  the  quality  of  the  research  should  be  improved. 

Such  a  procedure  is  not  required  by  NIH,  but  some  branch 
chiefs  do  this  to  manage  their  research  work. 

We  agree  with  HEW  that  early  termination  of  certain 
projects  indicates  a  strength  in  the  review  of  ongoing 
research  by  some  institutes.  For  example,  some  projects 
were  terminated  early  when  the  board  of  scientific  counselors 
found  the  research  to  be  of  low  scientific  merit  or  not 
directly  related  to  the  institute's  mission.  However,  only 
2  of  the  3  institutes  were  reviewing  specific  ongoing  projects 
through  their  boards.  We  believe  that  such  reviews  should 
be  performed  in  all  institutes. 

After  considering  HEW's  comments,  we  continue  to  believe 
our  proposals  have  merit  and  would  benefit  research.  We  do 
not  believe  it  would  be  as  costly  as  HEW  indicates,  since  its 
comments  assumed  an  outside  peer  review  group  would  be  needed. 
In  our  opinion,  a  peer  review  group  of  NIH  scientists  could 
be  used  for  such  a  purpose. 

RECOMMENDATIONS 


We  recommend  that  the  Secretary,  HEW,  instruct  the 
Director,  NIH,  to  require  that  written  plans  for  all  in¬ 
tramural  research  be  reviewed  and  approved  by  peer  review 
groups  before  research  projects  are  initiated  and  that 
reviews  be  made  by  boards  of  scientific  counselors  of  all 
ongoing  research  projects. 
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CHAPTER  5 


SCOPE  OF  REVIEW 


We  made  our  review  at  National  Institutes  of  Health 
headquarters  in  Bethesda,  Maryland,  and  at  National  In¬ 
stitute  of  Environmental  Health  Sciences,  Research  Triangle 
Park,  North  Carolina.  We  reviewed  3  of  the  10  research 
institutes  existing  at  the  time  of  our  review:  National 
Institute  of  Child  Health  and  Human  Development,  National 
Institute  of  Allergy  and  Infectious  Diseases,  and  NIEHS. 

During  our  review,  the  National  Institute  of  Aging,  the 
11th  NIH  institute,  was  established  from  components  of 
NICHD  that  had  been  doing  research  on  aging.  These 
tranches  of  NICHD  and  related  research  grants  are  not  dis¬ 
cussed  in  this  report. 

We  contacted  officials  within  each  institute  as  well  as 
at  the  NIH  directorate.  We  also  reviewed  the  policies, 
procedures,  and  implementing  regulations  concerning  intra¬ 
mural  research  and  research  grants  in  the  extramural  programs 
to  determine  how  well  these  programs  were  being  administered. 
In  addition,  we  obtained  documentation  from  agency  files 
relative  to  individual  research  grants  and  intramural  research 
projects  funded  in  fiscal  years  1973  and  1974. 
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DEPARTMENT  OF  HEALTH,  EDUCATION,  AND  WELFARE 

OFFICE  OF  THE  SECRETARY 
WASHINGTON.  D  C-  20201 


MAR  17  1976 


Mr.  Gregory  J.  Ahart 
Director ,  Manpower  and 
Welfare  Division 
United  States  General 
Accounting  Office 
Washington,  D.C.  20548 

Dear  Mr.  Ahart: 

The  Secretary  asked  that  I  respond  to  your  request  for 
our  comments  on  your  draft  report  entitled,  "Better 
Controls  Needed  Over  Biomedical  Research  Supported  by 
the  National  Institutes  of  Health. "  The  enclosed  comments 
represent  the  tentative  position  of  the  Department  and  are 
subject  to  reevaluation  when  the  final  version  of  this 
report  is  received. 

We  appreciate  the  opportunity  to  comment  on  this  draft 
report  before  its  publication. 

Sincerely  yours. 


:  \  .  . 

V  K  ( 

John  D.  ^oung 

Assistant  Secretary,  Comptroller 


Enclosure 
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COMMENTS  OF  THE  DEPARTMENT  OF  HEALTH,  EDUCATION,  AND  WELFARE  ON  THE 
COMPTROLLER  GENERAL'S  DRAFT  REPORT  TO  THE  CONGRESS  OF  THE  UNITED  STATES 
ENTITLED  "BETTER  CONTROLS  NEEDED  OVER  BIOMEDICAL  RESEARCH  SUPPORTED  BY 
THE  NATIONAL  INSTITUTES  OF  HEALTH" 


General  Comments 


The  administrative  practices  of  almost  any  enterprise  can  be  strength¬ 
ened  or  improved.  NIH  is  no  exception.  However,  while  it  is  frequently 
easy  to  design  apparently  appropriate  improvements,  particular  care  must 
be  taken  that  instituting  such  improvements  does  not  irreversibly  alter 
the  integrity  of  the  involved  organization. 

There  are  two  categories  of  recommendations  presented  in  the  GAO  draft 
report.  In  one  category  are  those  which  recommend  strengthen! ng  current 
administrative  practices.  We  have  generally  concurred  with  these  recom¬ 
mendations.  In  the  second  category  are  those  which  recommend  major  alter¬ 
ations  in  NIH  program  practices  and  philosophies.  We  do  not  concur  with 
implementing  these  recommendations.  We  believe  that  implementing  them 
would  be  extremely  costly  with  little  prospect  of  appreciable  benefits 
and  might  seriously  damage  both  the  NIH  extramural  and  intramural  programs. 

The  National  Institutes  of  Health  is  recognized  internationally  as  one  of 
the  preeminent  biomedical  research  institutions  in  the  world.  During 
Fiscal  Year  1975,  NIH  paid  about  38  percent  of  the  cost  of  all  health 
related  research  and  development  performed  in  the  United  States.  Most 
of  the  research  funded  by  NIH  was  performed  under  grants  and  contracts 
awarded  to  academic  and  scientific  institutions  located  throughout  the 
United  States  and  even  in  many  foreign  countries.  The  remainder  was  per¬ 
formed  in  NIH  laboratories . 

During  the  past  two  decades  of  NIH  leadership  in  the  fields  of  medicine 
and  related  sciences,  the  majority  of  the  Nobel  Prizes  in  physiology  or 
medicine  have  been  awarded  to  United  States  scientists  supported  at 
least  partly  by  NIH.  In  addition,  since  1968,  three  NIH  intramural 
scientists  have  received  the  Nobel  Prize. 

During  this  period  NIH  has  established  a  positive  and  effective  relation¬ 
ship  with  the  academic  and  scientific  community  which  houses  most  United 
States  biomedical  researchers.  This  relationship  has  encouraged  maximum 
research  productivity  with  a  minimum  of  administrative  direction  and  con¬ 
trol.  Under  this  close  working  rel ati onshi p ,  NIH  has  been  able  to  support 
the  development  of  needed  advances  in  basic  knowledge  and  health  technol¬ 
ogy  while  academic  institutions  have  remained  strong  in  their  primary 
functions.  These  institutions  have  also  expanded  their  capacity  and 
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expertise  to  conduct  biomedical  research  and  have  exploited  the  important 
role  which  research  plays  in  enhancing  the  quality  of  health  professional 

education. 

The  high  quality  of  the  NIH  extramural  program  has  primarily  resulted 
from  the  intensive,  incisive  review  of  grant  applications  under  the  peer 
review  system.  Under  the  intramural  program,  the  high  quality  of  the 
research  has  resulted  from  the  careful  selection  of  staff  scientists  who 
are  given  maximum  freedom  to  develop  their  creative  ideas  subject  to  con¬ 
tinuing  review  and  evaluation  by  NIH  management  and  their  peers  from  both 
within  and  outside  NIH. 

Following  are  comments  on  the  specific  recommendations  presented  in  the 
GAO  draft  report. 

GAO  Recommendations 


The  Secretary  of  HEW  should  require  the  Director  of  NIH  to  develop  a  sys¬ 
tem  to  identify  noncompeting  research  grants  with  less  scientific  merit 
than  approved  unfunded  competing  applications  and  to  make  an  assessment 
as  to  which  grants  should  be  funded. 

PHEW  Comments 

We  do  not  concur  in  this  recommendation.  (1)  Not  funding  selected  non¬ 
competitive  grants  to  provide  funds  for  awarding  new  applications  would 
be  both  impractical  and  wasteful.  (2)  Attempting  to  establish  and  operate 
the  system  necessary  for  identifying  ongoing  projects  for  termination 
would  require  large  staff  increases.  (3)  Cutting  off  funds  for  selected 
noricompeti ng  grants  would  be  very  disruptive  to  the  stability  of  the  scien 
tific  community.  These  points  are  elaborated  on  in  the  following. 

In  view  of  the  unknowns  involved  in  all  research  ventures,  the  subjective 
nature  of  priority  scores,  and  the  loss  of  resources  devoted  to  approved, 
ongoing  projects  if  they  are  prematurely  terminated,  there  appears  to  be 
no  justification  for  terminating  such  projects  to  provide  funds  for  new 
projects  with  better  priority  scores.  To  do  so  would  result  in  a  failure 
to  honor  previous  commitments,  would  be  unsound  because  projects  would  be 
terminated  before  allowing  time  for  fruition,  and  would  destroy  the  faith 
of  scientists  and  their  institutions  in  the  stability  of  Federal  biomed¬ 
ical  research  programs. 

Grants  are  made  by  NIH  to  support  biomedical  research  to  be  performed  by 
scientists  employed  elsewhere.  Each  year,  over  10,000  research  and  sev¬ 
eral  thousand  other  grant  applications  are  submitted  to  NIH.  From  these. 
Nil!  must  select  those  most  likely  to  succeed  and  most  relevant  to  its 
program  objectives.  To  assure  that  the  selection  is  based  upon  the  best 
expertise  available,  NIH  established  the  grants  peer  review  system. 
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This  system  consists  of  many  discipline-oriented  initial  review  groups 
generally  comprised  of  12  to  18  experts,  hired  by  NIH  as  temporary  con¬ 
sultants,  and  a  national  advisory  council  or  board  for  each  Bureau, 
Institute,  or  Division  of  NIH  which  awards  grants.  The  initial  review 
groups  consider  the  scientific  and  technical  merit  of  each  grant  appli¬ 
cation  and  render  their  recommendations  and  technical  conmentaries  to  the 
appropriate  advisory  council  or  board,  which  provides  final  recommenda¬ 
tions  to  NIH  staff.  NIH  staff  may  fund  only  grant  applications  which 
were  approved  by  the  councils  or  boards.  This  dual  peer  review  system 
provides  a  scope  and  variety  of  expertise  in  all  fields  of  biomedical 
research  that  NIH  could  not  effectively  and  economically  maintain  on  its 
staff. 

Members  of  the  review  groups  make  subjective  judgments  of  each  grant  appli¬ 
cation.  They  first  judge  whether  the  application  has  sufficient  merit  for 
NIH  support.  If  it  does,  they  recommend  approval;  if  not,  disapproval. 
Therefore,  all  approved  applications  merit  funding  if  sufficient  funds  are 
available.  TFe  reviewers  assign  numerical  values  to  each  application  to 
reflect  their  perceptions  of  its  merit.  Values  are  assigned  on  a  scale 
of  1  to  5  with  1  being  the  highest  priority.  The  assigned  numbers  are 
averaged  to  represent  a  consensus  judgment  rather  than  any  one  individual's 
judgment. 

The  GAO  draft  report  reflects  the  attitude  that  the  priority  score  is  an 
exact  indicator  of  quality  and  that  small  differences  reflect  real  differ¬ 
ences  in  scientific  merit.  This  is  not  true.  Real  differences  in  merit 
arc  probably  reflected  by  differences  of  30  or  more  points  in  the  prior¬ 
ity  scores.  Differences  in  the  scores  clustered  around  the  median  prob¬ 
ably  are  not  significant  because  they  are  an  average  of  judgments  of 
individuals  with  differing  opinions  and  degrees  of  expertise  on  the  spe¬ 
cific  subjects  presented  in  the  applications.  Therefore,  we  cannot 
emphasize  too  strongly  that  the  priority  score  can  be  used  only  as  a  tool 
and  should  not  be  looked  upon  as  a  precise  measurement.  Clearly,  small 
differences  in  priority  scores  should  not  be  used  as  a  basis  for  termi¬ 
nating  an  ongoing  grant  awarded  previously  in  favor  of  awarding  a  new  one. 

Priority  scores  are  only  one  factor  considered  by  NIH  in  selecting  which 
grunt  applications  to  fund.  All  applications  recommended  for  approval  by 
the  councils  or  boards  merit  funding.  Many  factors  influence  which 
approved  applications  are  selected  for  funding.  Among  these  are: 

(1)  the  relative  scientific  merit  of  a  project,  as  indicated  by  its 
priority  score  together  with  the  detailed  critique  of  it  which  was  pre¬ 
pared  by  the  initial  peer  review  group;  (2)  the  importance  of  a  particu¬ 
lar*  project  to  the  program  objectives  of  the  appropriate  Institute  or 
Division;  (3)  the  need  to  protect  previous  investments  in  meritorious 
ongoing  research,  as  determined  by  current  peer  reviews;  (4)  the  need  to 
provide  opportunities  for  new  research  and  researchers  to  enter  the  sys¬ 
tem;  and  (5)  the  amount  of  funds  available. 
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Under  the  NIH  peer  review  system  the  scientific  merit  of  competing 
research  grant  proposals  and  the  capabilities  of  the  respective  inves¬ 
tigators  are  incisively  examined.  Decisions  to  support  proposed  research 
are  at  best  "judgment  calls,"  not  guarantees  of  success  or  even  of 
research  product! vi ty . 

Most  scientists  agree  that  it  takes  approximately  two  to  three  years  for 
most  projects  to  be  developed  enough  to  produce  the  kind  of  tangible 
results  necessary  to  evaluate  progress.  That  the  average  period  of  sup¬ 
port  recommended  by  the  peer  review  groups  is  about  three  years  attests 
to  this  judgment.  In  most  instances,  sufficient  evidence  would  not  be 
available  in  less  time  on  which  to  make  intelligent  funding  decisions. 
Because  it  takes  about  eight  months  for  a  project  to  be  reviewed,  an 
award  has  to  be  for  about  three  years  if  a  funding  hiatus  is  to  be 
avoided  while  a  renewal  application  is  under  review. 

Research  does  not  progress,  at  a  constant  rate  with  progress  continuing 
in  a  straight-line  fashion  commencing  with  the  first  day  of  funding. 

During  the  earliest  phases  of  a  research  project  and  up  until  the  period 
when  data  analysis  is  fairly  well  advanced,  there  may  be  no  more  to  judge 
than  the  basic  considerations  that  were  available  to  the  initial  review 
group .  Much  research  which  ultimately  proves  to  be  of  the  highest  value 
in  advancing  medical  technology,  such  as  research  on  the  tissue  culture 
of  disease-causing  viruses,  has  been  very  difficult  to  accomplish  and 
required  many  years  to  bring  to  fruition. 

The  GAO  report  places  great  importance  on  the  fact  that  a  "considerable" 
portion  of  ongoing  grants  are  not  funded  again  when  they  are  submitted 
for  renewal.  Because  a  grant  is  not  funded  as  a  competing  renewal  does 
not  mean  that  it  should  not  have  been  funded  originally  or  that  it  should 
have  been  terminated  earlier.  If  projects  were  always  successful,  they 
would  not  be  research  projects.  In  resear:h  areas  failures  or  only  par¬ 
tial  successes  have  to  be  expected.  Adequate  time  must  be  provided  to 
allow  a  project  to  progress  to  the  point  when  its  relative  merit  and 
success  can  be  evaluated.  If  adequate  time  is  not  provided,  resources 
devoted  to  the  project  would  be  wasted  and  those  scientific  breakthroughs 
that  are  realized  only  near  the  end  of  grant  periods  would  be  precluded. 
There  is  no  adequate  basis  in  NIH  experience  for  suggesting  that  termi¬ 
nating  projects  after  only  a  year  or  two  to  fund  new  projects  with  better 
priority  scores  is,  in  fact,  realistic. 

The  review  of  competing  renewal  applications  is  carried  out  with  the  same 
expert  review  and  deliberations  as  are  made  when  selecting  new  grants  for 
funding,  except  that  there  is  also  a  research  record  which  can  be  eval¬ 
uated.  Since  an  investigator  working  on  a  project  funded  for  three  years 
must  submit  a  renewal  application  for  competitive  review  about  one  year 
before  the  first  project  period  terminates,  accomplishments  during  the 
I irst  two  years  of  the  project  are  available  to  the  initial  review  groups. 
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A  1974  study  by  the  Rand  Corporation  showed  that  progress  on  ongoing 
projects  is  fully  considered  by  the  review  groups  when  renewal  requests 
are  reviewed. 

The  statement  on  page  7*  of  the  GAO  report  that  180  approved  but  unfunded 
competing  renewals  did  not  have  sufficient  scientific  merit  to  warrant 
(underlining  added)  funding  by  the  respective  institutes  is  misleading. 

If  a  grant  is  recommended  for  approval,  by  definition,  it  warrants  fund¬ 
ing.  The  fact  is  that  there  were  not  adequate  funds  to  permit  their 
funding.  More  than  90  percent  of  the  competing  renewals  were  considered 
of  sufficient  merit  to  warrant  funding.  This  is  a  significantly  higher 
approval  rate  than  exists  for  new  grant  requests,  confirming  that  the 
initial  review  of  new  proposals  screens  out  most  of  those  not  likely  to 
succeed. 

In  1964,  the  Wooldridge  Correnittee,  one  of  the  many  outside  groups  to 
evaluate  the  NIH  peer  review  system,  reported  that  the  peer  review  system 
was  about  97  percent  accurate.  Furthermore,  the  original  findings  of  the 
Wooldridge  Cornu i ttee  have  been  substantiated  by  the  1974  study  of  the  NIH 
research  grant  peer  review  system  by  the  Rand  Corporation,  which  found  a 
very  good  correlation  between  the  productivity  of  NIH  supported  research 
and  the  original  review  judgments. 

NIH  scientist  administrators  do  use  annual  progress  reports  as  well  as 
publications  to  review  the  progress  of  grant-supported  projects.  Generally, 
progress  reports  for  regular  research  grants  are  reviewed  primarily  to 
determine  whether  the  research  is  focused  on  the  original ly-approved  sub¬ 
ject.  This  review  may  not  be  highly  detailed  because  of  the  heavy  work¬ 
load  of  NIH  scientist  admi ni strators . 

Grants  for  centers  and  some  program  projects  are  intended  to  support  long¬ 
term,  often  multidisciplinary  research  and  development.  Such  activities, 
involving  teams  of  researchers  in  both  basic  and  clinical  sciences, 
require  longer  periods  to  become  fully  operational.  Accordingly,  the  NIH 
Institutes  have  developed  various  special  procedures  for  monitoring  scien¬ 
tific  progress  and  the  quality  of  the  research,  involving  periodic  visits 
by  NIH  staff  and  consultants,  workshops  of  center  directors  and  their 
research  staff,  and  local  advisory  groups  to  oversee  center  activities. 

A  full  NIH  peer  review  is  often  scheduled  at  the  end  of  the  third  year, 
for  a  five-year  program,  to  assure  that  the  investigators  are  told  of  any 
need  for  improvement  or  modification  well  before  the  end  of  the  project 
period. 

If  it  were  to  attempt  to  monitor  grantee  progress  and  evaluate  and  judge 
that  progress  with  a  view  toward  terminating  or  withholding  funding  on 
less  worthy  existing  projects,  NIH  would  have  to  institute  similar  sys¬ 
tems  within  the  granting  Institutes  and  Divisions.  This  would  require  a 

*GAO  note:  Page  number  referred  to  by  HEW  has  been  changed  to 
reflect  page  number  in  this  report. 
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tremendous  commitment  of  manpower  that  NIH  does  not  now  possess.  The 
manpower  dedicated  to  performing  this  function  would  need  the  quantity, 
quality,  and  variety  of  expertise  now  contained  within  the  peer  review 
system.  Existing  extramural  scientific  staff  could  not  conduct  in-depth 
reviews  of  scientific  progress  on  the  14,000  active  research  projects 
even  if  it  were  assumed  that  such  progress  was  conclusive  at  the  time  of 
rev i ew . 

Since  few  projects  can  real  istical ly  be  evaluated  and  judged  after  only 
a  year's  operation,  it  does  not  appear  that  many  projects  would  be  iden¬ 
tified  for  termination  prior  to  the  end  of  their  project  periods.  There¬ 
fore,  large  commitments  of  scientific  manpower  to  attempt  to  make  such 
evaluations  and  judgments  are  not  justified.  Funded  projects  should  be 
allowed  to  run  for  the  periods  approved  by  the  peer  review  committees, 
unless  a  project  is  terminated  for  cause,  such  as  failure  to  perform  in 
accordance  with  the  terms  pf  the  grant  award.  This  leaves  the  decision 
on  a  project's  worth  within  the  peer  review  system  where  it  most  appro¬ 
priately  belongs. 

To  accomplish  NIH  extramural  research  objectives,  the  institutions  in 
which  the  research  is  conducted  must  be  strong  and  stable.  Unilateral 
termination  of  or  withholding  of  funds  for  ongoing  projects  would  create 
an  undesirable  instability  within  the  biomedical  community,  thus  destroy¬ 
ing  the  credibility  of  NIH  as  a  major  fund'ng  agency,  retarding  the  con¬ 
duct  of  research  in  these  institutions,  and,  most  importantly,  bringing 
about  an  unnecessary  tension  in  our  relationships  with  the  scientific 
community  upon  which  we  depend  for  research  progress  and  medical  advances. 

GA0_  Recomme n d ations 

The  Secretary  of  HEW  should  require  the  Director  of  NIH  to  establish 
specific  guidelines  delineating  what  should  be  included  in  the  annual 
scientific  progress  report  and  instructions  to  grant  administrators  on 
their  review  of  non-competing  grants.  Written  comments  on  such  reviews 
should  be  incorporated  into  the  official  grant  files. 

DHEW  Comments 

We  concur  in  part  with  the  GAO  recommendations  as  follows:  For  centers 
and  similar  large  and  multifaceted  grants,  most  NIH  programs  now  require 
very  detailed  annual  progress  reports  prepared  to  exacting  specifications. 
These  are  reviewed  in  detail.  Such  grants  are  often  also  subjected  to 
surveillance  by  special  program  advisory  committees  or  a  subcommittee  of 
the  appropriate  Institute's  national  advisory  council  or  board.  Visits 
by  program  staff,  often  with  consultant  experts,  also  track  process  and 
assure  that  the  programs  continue  to  meet  the  objectives  original ly 
approved . 
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For  regular  research  project  grants  and  any  center  or  other  large  grants 
for  which  there  may  not  be  current  guidelines  for  annual  reporting  or 
annual  staff  review,  NIH  will  survey  the  use  of  structured  progress 
reports  and,  if  necessary,  will  develop  instructions  for  grantees  to  use 
in  preparing  such  reports. 

We  agree  that  there  is  a  lack  of  formalized  NIH-wide  instructions  given 
to  NIH  start  for  reviewing  progress  reports  arid  taking  appropriate 
actions  if  the  material  presented  in  a  report  indicates  that  problems 
exist.  Recognizing  that  individual  NIH  institutes  follow  their  own  pro¬ 
cedures  in  reviewing  progress  reports,  NIH  will  develop  appropriate  NIH- 
wide  instructions.  However,  it  must  be  recognized  that  research  does 
not  progress  evenly  and  that  success  or  failure  is  not  normally  achieved 
within  the  span  of  only  a  few  years.  NIH  scientific  staff  resources  are 
limited.  Therefore,  it  is  not  possible  for  NIH  scientist  administrators, 
responsible  each  year  for  about  14,000  projects,  to  determine  the  merit 
of  scientific  progress  through  reviewing  annual  progress  reports .  How¬ 
ever,  they  can  assure  that  the  research  continues  within  the  scope  of 
the  original  project  and  that  the  investigator  has  shared  with  NIH  find¬ 
ings  made  during  the  previous  year. 

Preparing  written  comments  each  time  a  progress  report  is  reviewed  would 
create  a  large  workload  for  an  already  overourdened  program  staff.  We 
see  no  need  for  or  appreciable  benefits  to  be  gained  from  staff  preparing 
such  written  comments  unless  they  have  determined  that  problems  exist. 
While  program  staff  already  document  problems  they  have  identified  and 
resulting  actions,  we  will  instruct  that  grant  files  should  be  appropri¬ 
ately  documented  when  problems  are  encountered. 

GAO  R e c omm e nd a  tj  on 

The  Secretary  of  HEW  should  require  the  Director  of  NIH  to  clarify  the 
specific  authority  of  grant  admi ni stra tors  to  withhold  funds  and  dissem¬ 
inate  such  information  to  all  research  grant  officials. 

DHEW  Comments 

We  concur  with  the  recommendation.  The  Division  of  Grants  and  Contracts, 
Office  of  the  Assistant  Secretary  for  Health,  is  developing  guidelines 
in  this  area  for  use  throughout  the  Public  Health  Service  (PHS). 

However,  NIH  does  have  certain  authority  to  withhold  funds.  The  GAO 
report  states  that  NIH  does  not  have  the  authority  to  terminate  a  grant 
for  "poor  research  progress"  and  that  NIH  does  not  have  the  authority  to 
terminate  a  grant  on  the  basis  of  "public  interest  or  convenience  of  the 
Government."  We  feel  that  these  statements  are  erroneous.  The  DHEW 
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Grants  Administration  Manual  (GAM),  chapter  DHEW:  1-85  permits  the 
grantor  to  unilaterally  terminate  the  support  of  a  project  when  the 
grantee  has  "failed  to  show  satisfactory  progress"  and  for  other  reasons 
"not  necessarily  within  the  grantee’s  control,"  such  as  a  determination 
that  the  research  is  no  longer  of  public  benefit.  Under  Title  45  CFR 
Part  74  and  GAM,  chapters  DHEW:  1-85  and  PHS:  1-500,  the  Director,  NIH, 
may  stop  the  support  of  a  research  grant  if  he  determines  that  the  grantee's 
progress  does  not  materially  conform  with  the  terms  and  conditions  of  the 
grant  award  or  the  direction  of  the  research  has  been  so  altered  that  it 
no  longer  is  heal th-related  or  of  public  benefit. 

The  report  also  states  that  "Although  NIH  can  withhold  funds  from  a 
grantee  for  the  remainder  of  an  approved  project  period,  it  cannot  rescind 
funds  that  were  approved  but  unexpended  for  the  earlier  part  of  the  grant 
period."  NIH  cannot,  except  for  cause,  rescind  funds  that  have  been 
approved  but  unexpended  during  the  budget  period  for  which  they  were 
awarded.  However,  since  NIH  must  approve  all  carry-overs  of  funds  from 
one  budget  period  to  the  next,  it  can  prevent  a  grantee  from  carrying 
over  unexpended  funds.  When  NIH  approves  carrying  over  unexpended  funds 
it  generally  uses  them  to  reduce  future  awards. 

GAO  Re  comm  e  n_d  a  tion 

The  Secretary  of  HEW  should  require  the  Director  of  NIH  to  change  the 
regulations  for  terminating  research  grants  to  permit  selective  termina¬ 
tion  of  grants  in  cases  where  (1)  there  is  poor  progress,  (2)  better 
applications  could  be  funded,  or  (3)  the  research  is  no  longer  of  public 
benef i t . 

DHEW  Comments 

We  disagree  with  the  recommendation.  We  eel  that  the  existing  research 
grant  regulations,  together  with  established  DHEW  and  PHS  policies,  are 
sufficient  to  deal  witti  those  situations  where  it  is  appropriate  to  stop 
the  support  of  a  project. 

Also,  we  do  not  believe  that  situations  dictating  the  need  for  terminat¬ 
ing  a  grant  occur  very  often  within  NIH's  extramural  program.  By  limit¬ 
ing  most  research  projects  to  about  three  years  of  support  between 
competitive  reviews,  by  applying  incisive  peer  review  to  all  research 
grant  applications,  and  by  carefully  selecting  for  funding  only  approved 
projects  having  potential  benefit,  NIH  assures  that  research  of  poor 
quality  or  lacking  relevance  to  important  hea 1th  problems  is  seldom 
encountered  in  its  grant  activities. 

GAO  Recommendation 

The  Secretary  of  HEW  should  require  the  Director  of  NIH  to  require  all 
institutes  to  use  "normalized"  priority  scores  in  funding  research  grants. 
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DHEW  Comments 


We  concur  in  the  GAO  recommendation  to  the  extent  that  we  will  explore 
further  the  matter  of  using  normalized  scores.  Between  1971  and  1973  an 
NIH  Committee  reviewed  and  developed  a  normalization  procedure  for  assign¬ 
ing  priority  scores  which  was  put  into  effect  for  one  year.  Under  the 
assumption  that  all  fields  of  science  are  essentially  of  equal  merit, 
the  normalization  computation  is  intended  to  adjust  for  assumed  varia¬ 
tions  in  the  behavior  of  different  review  groups  in  rating  applications. 
Because  this  assumption  is  only  partly  true,  the  NIH  Executive  Committee 
for  Extramural  Affairs  elected  to  retain  the  use  of  raw  scores  but  to 
also  make  normalized  scores  available  as  an  aid  to  the  institutes.  No 
priority  score  system,  whether  normalized  or  not,  can  replace  staff  and 
councils'  consideration  of  each  application  in  the  "gray  zone"  where 
normalization  may  impact  upon  awarding  decisions. 

Although  the  normal ization  process  has  been  extensively  studied  in  the 
past,  we  will  explore  it  further  by  referral  to  the  NIH  Grants  Peer  Review 
Study  Team  which  was  recently  established  by  the  Director,  NIH,  to  exam¬ 
ine  all  aspects  of  the  NIH  peer  review  system. 

GAO  Recommendati ons 

The  Secretary  of  HEW  should  direct  the  Director  of  NIH  to  (1)  require 
all  institutes  to  more  closely  monitor  the  submission  of  required  reports 
when  a  grant  expires,  (2)  prohibit  the  acceptance  of  terminal  progress 
reports  which  do  not  cover  the  entire  period  of  grant  support,  (3)  pro¬ 
hibit  the  funding  of  researchers  when  they  are  known  to  be  in  violation 
of  the  terms  and  conditions  of  previous  NIH  support,  and  (4)  establish 
an  information  system  capable  of  exchanging  information  on  delinquent 
research  grant  reports  between  institutes. 

DHEW  Comments 

We  partly  concur  in  the  GAO  recommendations,  as  follows: 

We  agree  that  required  terminal  reports  should  be  submitted.  The  DHEW 
Grants  Administration  Manual,  chapters  PHS:  1-462  and  1-42,  provides 
policy  guidance  on  grant  closeout  requirements  and  delinquent  grantee 
reporting  procedures,  respectively.  We  plan  to  take  steps  to  ensure 
the  timely  submission  of  terminal  progress  reports. 

We  agree  with  the  recommendation  that  terminal  reports  should  cover  the 
entire  grant  period. 

GAO  Rec  Qi  nmen  dat jo  ns 

The  Secretary  of  HEW  should  direct  the  Director  of  NIH  to  require  that 
written  protocols  for  all  intramural  research  projects  be  reviewed  and 
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approved  by  a  peer  review  group  prior  to  the  initiation  of  research 
projects  and  that  reviews  be  made  of  specific  ongoing  research  projects. 
If  augmented  by  ad  hoc  consultants,  the  Boards  of  Scientific  Counselors 
could  perform  this  function. 

BIIEV/  Comments 

1/e  do  not  concur  in  the  recommendation  that  intramural  research  projects 
be  reviewed  and  approved  by  a  peer  review  group  prior  to  the  initiation 
of  research  projects.  We  believe  that  the  present  internal  system  of  con 
trol  over  intramural  research  is  more  economical,  efficient,  and  timely 
than  a  peer  review  system  similar  to  or  duplicative  of  the  system  used 
for  awarding  grants  would  be.  The  quality  of  the  work  of  NIH  intramural 
scientists  as  well  as  the  priority  of  their  ongoing  research  projects  js_ 
already  routinely  reviewed  and  evaluated  at  several  levels  within  the 
Mill  system. 

Operating  the  type  of  formalized  peer  review  system  used  for  awarding 
NII1  grant  funds  is  very  costly  in  several  way s.  NIH  spends  considerable 
money  in  consultant  fees  and  travel  costs  for  the  hundreds  of  outside 
experts  it  uses  to  review  and  evaluate  grant  applications.  Much  high- 
level  scientific  manpower  is  spent  in  reviewing  and  evaluating  applica¬ 
tions,  making  needed  site  visits  to  prospective  grantees,  and  meeting 
three  times  yearly  to  decide  on  the  acceptability  of  grant  applications 
and  set  priorities  on  those  judged  acceptable. 

However,  a  system  had  to  be  established  to  assure  that  only  meritorious 
science  is  supported  under  the  NIH  grants  mechanism.  To  meet  this  need 
the  peer  review  system  has  evolved  as  the  best  mechanism  for  defining 
.•rid  administering  a  quality  extramural  biomedical  research  program 
which  is  spread  over  a  wide  geographic  area.  However,  it  is  not  the 
best  mechanism  for  administering  an  ongoing  intramural  scientific  pro¬ 
gram.  The  intramural  research  program  of  NIH  is  largely  self-contained 
in  the  geographic  sense  and  can  avail  itself  of  a  highly  appropriate  but 
different  mechanism  for  defining  and  administering  a  quality  in-house 
research  program,  namely  continuing  review  and  evaluation,  carried  out 
both  formally  and  informally  by  the  scientists'  colleagues  and  superiors. 

Only  if  the  operating  NIH  intramural  system  were  not  providing  highly 
productive,  high  level  research  results  would  it  appear  desirable  to 
consider  imposing  on  it  such  a  costly  mechanism  as  outside  peer  review. 

I  he  Nlll  intramural  program,  however,  by  any  method  of  measurement  yet 
devised,  has  produced  extremely  well,  resulting  in  very  high  quality 
scientific  endeavors. 

Added  to  the  above  is  our  fear  that  imposing  an  outside  peer  review 
system  could  have  serious  destructive  effects  on  the  NIH  intramural 
research  program.  NIH  has  two  assets  that  help  it  hold  first-class 
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scientists.  They  are  (1)  a  relative  freedom  to  pursue  research  activ¬ 
ities,  without  submitting  formal  research  grant  applications  and  (2)  gen¬ 
erally  good  research  facilities.  Universities  typically  can  outbid  NIH 
on  salaries,  fringe  benefits,  and  professional  status.  If  NIH  had  to 
put  its  scientists  through  the  same  procedures  that  university  scientists 
go  through  to  obtain  grants,  we  could  lose  many  of  our  top  investigators. 

If  we  lost  many  top  investigators  because  they  no  longer  had  the  admin¬ 
istrative  and  continuity  benefits  present  at  NIH  to  offset  other  benefits 
available  elsewhere,  many  young  scientists  who  come  to  NIH  to  work  with 
and  learn  from  such  investigators  would  not  come  here.  If  these  occurred. 
NIH's  prestige  would  rapidly  decline  and  we  would  have  destroyed  a  scien¬ 
tific  system  and  organization  that  has  been  the  heart  of  biomedical 
research  and  training  in  the  United  States. 

The  following  paragraphs  discuss  in  more  detail  our  position  on  these 
GAO  recommendations. 

Intramural  NIH  scientists  are  under  the  control  and  supervision  of  care¬ 
fully  selected  section  and  laboratory  or  branch  chiefs  who  continually 
review  and  evaluate  their  work.  The  laboratory  or  branch  chiefs  report 
to  and  are  under  the  control  of  their  institutes'  scientific  directors 
who  report  to  their  institute  directors  and  also  to  the  NIH  Deputy  Direc¬ 
tor  for  Science,  both  individually  and  collectively,  at  semi-monthly 
Scientific  Directors'  Meetings. 

Every  time  a  technician  is  assigned  or  a  piece  of  equipment  purchased  a 
decision  is  made  on  the  quality  and  priority  of  the  research.  The  real 
reviews  and  decisions  are  made  within  the  area  of  allocating  scarce 
resources.  When  projects  do  not  measure  up,  such  resources  are  withheld 
or  negotiated  downward. 

Each  institute  has  a  Board  of  Scientific  Counselors,  comprised  of  scien¬ 
tists  from  outside  NIH,  which  reviews  in  an  advisory  capacity  the  insti¬ 
tute's  intramural  research  programs.  These  Boards  were  created  to  bring 
to  bear  on  the  various  intramural  programs  "he  direct  views  of  experts 
from  outside  NIH.  The  Boards,  in  essence  boards  of  visitors,  were  intended 
to  (1)  advise  Institute  Directors  and  their  Scientific  Directors  on 
research  program  designs,  emphases,  and  directions  and  (2)  review  and  gen¬ 
erally  evaluate  scientific  work  under  the  research  programs. 

NIH  has  considered  conducting  peer  reviews  of  proposed  research  projects 
by  the  members  of  the  Boards  of  Scientific  Counselors.  However,  we  feel 
that  this  is  an  i nappropr i ate  function  for  them.  We  believe  that  they 
can  best  be  used  to  provide  evaluations  of  ongoing  research  programs  and 
that  they  should  indicate  their  views  on  how  such  programs  fit  into  our 
overall  mission. 
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The  continuing  review  and  evaluation  mechanism,  such  as  used  at  NIH,  has 
gained  long  standing  acceptance  in  industry,  commerce,  and  academe 
because  it  is: 

(1)  continuous  rather  than  episodic; 

(2)  immediately  responsive  in  allocating  resources  in  the  continuous 
competition  for  the  limited  budget  and  space  available;  and 

(3)  dynamic  and  produces  viable  and  effective  feedback. 

An  important  contrast  can  be  made  between  the  extramural  peer  review 
process  and  the  intramural  review  and  evaluation  process.  In  the  former, 
there  is  only  periodic  opportunity  for  assessment  of  the  value  of  research 
work  being  proposed.  In  the  latter,  there  is  close  supervision  and  con¬ 
tinuous  assessment  by  superiors,  colleagues,  and  associates  during  the 
conduct  of  the  research  itself.  Therefore,  we  believe  that  the  continuing 
review  and  evaluation  mechanism  is  the  best  method  for  supporting  and 
maintaining  the  quality  of  the  NIH  intramural  program. 

Within  intramural  NIH,  the  selection  and  hiring  process  is  an  important 
part  of  the  management  and  direction  of  research.  For  this  reason  the 
NIH  intramural  program  has  developed  "add-on"  personnel  systems,  such  as 
the  Staff  Fellow  program.  These  programs  stand  at  the  heart  of  the  con¬ 
tinuing  review  and  evaluation  mechanism  described  above.  These  programs 
provide  an  opportunity  to  assess  and  measure  the  intellectual  excellence 
of  young  investigators  prior  to  conferring  tenure  in  the  form  of  a  per¬ 
manent  appointment. 

In  the  case  of  staff  fellows,  who  serve  in  a  non-tenured  capacity  for 
four  to  five  years,  NIH  has  the  opportunity  to  evaluate  their  produc¬ 
tivity  over  an  extended  period.  If  a  staff  fellow  does  not  develop  as 
anticipated  or  his  research  interests  are  not  congruent  with  those  of 
his  Institute,  his  appointment  can  be  terminated.  Thus,  NIH's  granting 
of  tenure  carries  with  it  the  commitment  to  support  a  scientist's  research 
so  long  as  he  is  productive.  Civil  Service  and  Commissioned  Officer  reg¬ 
ulations  limit  the  ability  to  terminate  the  employment  of  an  individual. 
Therefore,  NIH  puts  great  emphasis  on  the  initial  selection  process. 
Recommendations  for  conversion  to  permanent  status  are  made  to  the 
Deputy  Director  for  Science  by  the  Scientific  Directors  at  their  semi¬ 
monthly  meetings.  Such  conversion  is  a  highly  selective  process.  Only 
about  one-third  of  the  staff  fellows  who  completed  their  terms  of  appoint¬ 
ment  in  Fiscal  Year  1975  were  converted  to  permanent  status. 

Another  control  over  the  quality  of  NIH  scientists  is  the  review  process 
followed  when  a  permanent  scientist  is  proposed  for  promotion  to  the  next 
higher  grade  or  to  an  administrative  rank,  such  as  Section  Head  or  Lab¬ 
oratory  Chief.  Promotional  decisions  are  discussed  and  debated  carefully 
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at  the  semi-monthly  meetings  of  the  NIH  Scientific  Directors,  who  then 
recommend  action  to  the  Deputy  Director  for  Science. 

The  Scientific  Directors  take  these  responsibilities  very  seriously 
since  they  provide  the  major  elements  of  quality  control  across  the  intra¬ 
mural  programs  of  the  various  institutes  and  contribute  to  the  equity  and 
comparability  among  equivalent  scientific  ranks  at  NIH.  In  preparation 
for  bringing  their  nominations  to  the  Scientific  Directors'  meetings, 
individual  Scientific  Directors  thoroughly  study  and  investigate  the 
qualifications  of  the  scientists  in  question.  Inputs  into  a  decision 
on  nominating  an  NIH  scientist  either  for  conversion  to  permanent  status 
or  promotion  include  discussions  with  his  section  and  branch  chiefs;  an 
evaluation  of  his  publications  of  original  scientific  articles;  an  exam¬ 
ination  of  feedback  from  reviews  conducted  by  the  appropriate  Board  of 
Scientific  Counselors,  which  may  have  been  supplemented  by  ad  hoc  con¬ 
sultants;  a  review  of  his  performance  in  staff  seminar  presentations; 
and  a  review  of  other  materials  which  may  have  been  solicited,  such  as 
letters  from  eminent  scientific  leaders  outside  NIH. 

The  NIH  system  of  reviewing  the  quality  of  its  scientific  staff  places 
great  responsibility  on  each  Scientific  Director,  but  this  is  precisely 
his  responsibility  as  a  Federal  Science  Admi ni strator .  To  the  degree 
that  the  intramural  scientific  programs  have  been  of  high  quality  and 
producti vi ty ,  this  must,  in  large  part,  be  attributed  to  the  dedication, 
sensitivity,  and  judgment  of  the  Scientific  Directors. 

At  NIH,  a  key  aspect  of  research  management  is  the  high  level  of  the 
scientific  expertise  of  section  chiefs,  laboratory  or  branch  chiefs, 
and  institute  scientific  directors.  Research  seminar  programs  also 
serve  an  important  quality  control  function  at  NIH.  Such  seminars  of 
assembled  NIH  scientists  may  be  likened  to  the  site  visits  made  in  con¬ 
nection  with  grant  applications.  In  addition  to  continuing  in-house 
reviews,  the  boards  of  scientific  counselors,  bolstered  as  needed  by 
ad  hoc  experts,  periodically  review  in  detail  the  research  programs  of 
intramural  scientists. 

We  do  not  believe  that,  as  stated  in  the  GAO  report,  projects  being  ter¬ 
minated  indicates  a  weakness  in  the  NIH  system,  rather  partly  its 
strength.  By  the  very  nature  of  research,  not  all  projects  will  be  suc¬ 
cesses.  Failures,  or  less  than  total  successes,  have  to  be  expected. 

When  such  projects  can  be  identified  and  terminated,  the  system  must  be 
working  well.  However,  several  of  the  projects  cited  in  the  report  were 
terminated  because  of  shortages  of  resources  or  to  bring  certain  programs 
into  balance  with  others,  not  because  of  lack  of  scientific  excellence. 

In  the  face  of  limited  resources  we  must  continually  review  our  ongoing 
studies  to  insure  that  the  highest  priority  and  most  promising  efforts 
are  carried  out.  It  is  not  unusual  that  potentially  good  projects  are 
shelved  because  of  the  lack  of  resources.  Terminating  projects  does 
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not  mean  that  they  were  not  worthy  of  the  funding  they  had  already 
received.  Unfortunately ,  budget  constraints  which  prevent  funding 
worthwhile,  potentially  promising  projects  or  programs  can  cause  gaps 
in  our  scientific  knowledge.  Often  very  good  projects  may  be  shelved 
because  the  long-range  benefits  could  not  be  perceived  during  the  ini¬ 
tial  phases.  Given  additional  resources  we  would  fund  work  in  many 
additional  research  areas. 

hill  provides  an  environment  in  which  outstanding  scientists  can  pursue 
the  long-term  basic  and  clinical  research  needed  for  solutions  to  the 
more  difficult  scientific  problems.  This  is  an  environment  consisting 
of  an  unparalleled  number  of  scientists  in  many  disciplines  in  a  facility 
with  enormous  technological  depth  and  availability  of  specialized 
equipment. 

One  of  the  major  virtues  of  the  NIH  intramural  research  program  has  been 
its  ability  to  provide  the  environment  under  which  scientists  can  pursue 
important  long-term  research  studies  without  having  to  wrap  them  up  pre¬ 
maturely  because  of  arbitrary,  external  deadlines  imposed  by  limited- 
duration  grants.  Such  studies  would  be  difficult  to  pursue  under  the 
extramural  system.  Examples  of  these  are  the  long-term  studies  on  the 
so-called  "slow-virus"  diseases  and  longitudinal  studies  on  an  aging 
population.  Such  types  of  research  require  prolonged  commitment  of 
resources  and  a  faith  that  eventually  an  innovative  research  plan  will 
result  in  useful  knowledge.  The  intramural  program  has  been  able  to 
exploit  these  opportunities  because  of  the  flexibility  in  the  way  we 
administer  the  research  organization  and  the  fact  that  NIH  scientists 
are  protected  from  the  large  time  commitment  to  raising  money  which 
increasingly  faces  scientists  in  the  extramural  area. 

The  only  justification  for  materially  reorganizing  the  management  of 
Hill's  intramural  research  program  would  be  that  it  has  not  been  very 
successful.  However,  the  NIH  is  widely  regarded  as  providing  high  scien¬ 
tific  quality  research  influencing  very  importantly  the  directions  of 
biomedical  research  elsewhere.  It  serves  as  an  important  information 
and  consultation  center  and  as  an  important  training  institution  for  , 
young  scientists  in  basic  and  clinical  research.  Many  of  the  leaders  at 
the  NIH  and  in  biomedical  research  in  academic  institutions  get  their 
start  in  NIH  *  s  laboratories . 

By  implication,  the  GAO  recommendations  suggest  that  the  quality  of 
extramural  grant  projects  is  better  than  that  of  intramural  research 
projects  and  that,  if  intramural  projects  were  to  undergo  the  extramural 
peer  review  procedure  prior  to  their  initiation,  the  quality  of  such 
projects  would  be  improved  and  would  be  similar  to  that  of  grant  supported 
research. 
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In  fact,  the  general  quality  and  productivity  of  the  NIH  intramural 
research  programs  compare  very  favorably  with  research  conducted  extra- 
murally,  as  determined  by  several  objective  parameters  by  which  publica¬ 
tions  can  be  assessed. 

Computer  Horizons,  Inc.,  has  developed  a  weighted  formula  for  measuring 
the  influence  of  articles  published  in  scientific  journals.  Using  this 
formula  it  has  developed  and  analyzed  data  of  publications  by  NIH  scien¬ 
tists  in  the  biomedical  literature. 

Computer  Horizons  reported  that,  by  one  measure,  NIH's  publications  have 
a  higher  influence  per  publication  than  any  other  major  institution  in 
the  United  States  except  Harvard.  The  average  publication  weights  for 
the  15  leading  institutions  are: 

(1)  Harvard  University,  31.2, 

(2)  NIH,  29.8, 

(3)  University  of  Wisconsin,  28.2, 

(4)  Yale  University,  27.3, 

(5)  Johns  Hopkins  University,  24.9, 

(6)  University  of  Pennsylvania,  24.8, 

(7)  Columbia  University,  24.8, 

(8)  Stanford  University,  24.2, 

(9)  University  of  Washington,  23.5, 

(10)  University  of  California,  22.0, 

(11)  New  York  University,  21.9, 

(12)  State  University  of  New  York,  20.5, 

(13)  University  of  Michigan,  19.8, 

(14)  University  of  Minnesota,  19.8,  and 

(15)  Mayo  Foundation,  18.8. 

Furthermore,  its  analysis  of  240  institutions  showed  that  there  is  no 
other  substantial  institution  that  has  an  influence  per  publication 
figure  higher  than  NIH  except  the  Rockefeller  University,  which  has  an 
extraordi nari ly  high  publication  weight  of  42. 

Another  way  to  measure  intramural  NIH  productivity  is  to  compare  the 
percentage  of  the  members  in  the  6  societies  of  the  Federation  of  American 
Societies  for  Experimental  Biology  who  are  intramural  NIH  scientists 
(3.22%)  with  the  percentage  of  papers  from  the  United  States  which  were 
authored  by  intramural  NIH  scientists  and  were  published  in  various 
journals.  In  1973,  the  NIH  figures  were: 

(1)  for  the  1,000  biomedical  journals  studied,  3.43  ; 

(2)  for  the  20  biomedical  journals  that  have  the  highest 
influence  per  publication,  5.12%; 

(3)  for  the  20  biomedical  journals  with  the  highest  influence 
weight,  5.78  ; 
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(4)  for  12  biomedical  review  journals,  7,49%; 

(5)  for  the  Journal  of  Biological  Chemistry,  7.50%; 

(6)  for  Analytical  Biochemistry,  8.46%; 

(7)  for  the  Journal  of  Experimental  Medicine,  9.16%;  and 

(8)  for  the  Bio  physic  o~l  Jo  u  r  n  a  f,  11771 3%. 

Therefore,  N I H  appears  to  be  pulling  more  than  its  proportional  weight 
in  papers  published  in  the  more  prestigious  and  influential  scientific 
journals. 

N I H  has  considered  duplicating  within  the  intramural  research  program 
the  peer  review  system  used  to  evaluate  research  grant  proposals.  We 
concluded  that  imposing  such  a  system  on  the  intramural  program  would 
not  be  practical.  It  would  mean  that  all  intramural  research  projects 
would  have  to  be  reviewed  in  advance  by  panels  of  outside  experts.  Get¬ 
ting  enough  interested  and  competent  outsiders  to  review  all  the  projects 
of  a  full-time  staff  the  size  of  NIH's  would  be  extremely  costly,  both 
in  terms  of  the  consultant  fees  and  travel  costs  NIH  would  have  to  pay 
and  t fie  time  the  consultants  would  have  to  spend  making  such  reviews. 
Furthermore,  we  believe  it  would  be  impossible  to  administer  a  research 
program  like  ours  if  decisions  on  support  had  to  be  made  by  outside 
panels.  The  resources  we  deal  with  are  budgeted  positions,  space,  and 
money.  They  cannot  be  awarded  and  withheld  with  the  flexibility  of 
dollars  in  research  grants,  nor  can  decisions  on  projects  be  made  with¬ 
out  regard  for  the  impact  of  those  decisions  on  the  careers  of  the  scien¬ 
tists  in  a  full-time,  largely  tenured  service.  Finally,  and  most 
important,  an  outside  peer  review  of  projects  would  so  change  the  intra¬ 
mural  program  that  it  could  no  longer  provide  scientists  with  what  is 
the  unique  virtue  of  intramural  research,  the  opportunity  for  real  con¬ 
tinuity  of  effort  over  long  time  periods. 

Normally,  the  greatest  continuing  cost  element  in  research  today  is  the 
salaries  of  the  research  teams.  We  have  difficulty  envisioning  an  out¬ 
side  peer  review  system  which  could  respond  so  quickly  that  it  would 
not  result  in  intramural  investigators  and  their  staffs  awaiting  approval 
to  perform  their  work.  We  believe  such  a  system  would  be  neither  desir¬ 
able  nor  feasible. 

We  have  concluded  that  the  manpower  and  money  costs  of  imposing  an  out¬ 
side  peer  review  system,  plus  its  destructive  side  effects,  would  far 
outweigh  the  good  to  be  achieved  by  weeding  out  a  few  lower  priority 
projects.  There  is  no  good  evidence  to  indicate  that  the  savings  antic¬ 
ipated  from  pre-screening  intramural  research  projects  would  offset  the 
considerable  costs  involved  in  utilizing  such  a  procedure  in  a  research 
institution  the  size  of  Nil...  These  costs  would  involve  both  the  expenses 
associated  with  making  the  reviews  themselves  as  well  as  the  adverse 
impact  on  NIH  staff  caused  by  delays  in  initiating  projects  and  uncer¬ 
tainties  about  their  research  support. 


50 


APPENDIX  I 


APPENDIX  I 


NIH  is  a  first-class  research  institution.  Its  excellence  has  been 
recognized  with  three  Nobel  prizes  in  the  past  few  years.  The  high 
level  of  quality  will  be  difficult  to  maintain  during  a  time  of  no 
growth  or  actual  cutbacks.  It  can  be  done  with  strong  and  intelligent 
leadership.  We  do  not  see  how  mandatory  outside  peer  review  of  scien¬ 
tists  or  their  intramural  research  projects  would  benefit.  We  believe 
that  the  GAO  recommendation,  if  put  into  effect,  could  have  serious 
effects  on  the  NIH  intramural  research  program.  We  do  not  believe  that 
approval  of  a  written  protocol  prior  to  initiation  of  an  intramural 
research  project  would  be  a  desirable  procedure  to  impose  upon  NIH  intra¬ 
mural  projects.  Such  a  procedure  would  impose  an  unnecessary  degree  of 
formality  and  rigidity  to  the  ongoing  internal  management  of  the  intra¬ 
mural  research  program.  It  would  possibly,  by  stifling  work  in  particular 
areas,  actually  diminish  the  uniqueness  and  quality  of  the  intramural 
program. 
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PRINCIPAL  HEW  OFFICIALS  RESPONSIBLE  FOR 
ADMINISTERING  ACTIVITIES  DISCUSSED  IN  THIS  REPORT 


Tenure  of  office 
From  To 
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Aug . 

1975 
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Caspar  W.  Weinberger 

Feb . 
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Aug.  1975 
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1975 
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Charles  C.  Edwards 
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Feb . 
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July  1975 

Robert  S.  Stone 

May 

1973 

Jan.  1975 

DIRECTOR,  NATIONAL  INSTITUTE  OF 
ALLERGY  AND  INFECTIOUS  DISEASES: 
Richard  M.  Krause 

Nov . 

1975 

Present 

John  R.  Seal  (acting) 

Aug . 

1975 

Oct.  1975 

Dorland  J.  Davis 

Oct . 

1964 

Aug.  1975 

DIRECTOR,  NATIONAL  INSTITUTE  OF 
CHILD  HEALTH  AND  HUMAN  DEVELOP¬ 
MENT: 

Norman  Kretchmer 

Sept 

.  1974 

Present 

Gilbert  L.  Woodside  (acting) 

Sept 

.  1973 

Sept.  197 

DIRECTOR,  NATIONAL  INSTITUTE  OF 
ENVIRONMENTAL  HEALTH  SCIENCES: 
David  P.  Rail 

Mar. 

1971 

Present 

a/Acting  from  Feb. 


1975  until  May  1975. 
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